HPC:Conda
This section of the wiki discusses the use of Conda/Anaconda on the HPC.
Conda
Conda is a cross-platform, language-agnostic binary package manager. While using tools like Conda/Anaconda makes it easy for local package management and reusing Python virtual environments, invariably, it also allows most novice (and in some cases experienced) HPC users to inadvertently overwrite their .bashrc and/or .bash_profile files. Thus modifying crucial environment variables such as $PATH etc. We encourage users to tread with caution when using Conda.
Below is an example of a common issue that is caused by local Conda installations:
[asrini@consign ~]$ bsub -Is bash Job <64070984> is submitted to default queue <interactive>. <<Waiting for dispatch ...>> <<Starting on node082.hpc.local> (base) bash-4.2$ module avail module avail bash: module: command not found
The fix for the above to to make a copy of the Conda based .bashrc and .bash_profile files and then use the system default versions of these files and then re-apply only the non-Conda related changes in the now-older .bashrc/.bash_profile files:
[asrini@consign ~]$ mv $HOME/.bashrc $HOME/.bashrc_bak
[asrini@consign ~]$ mv $HOME/.bash_profile $HOME/.bash_profile_bak
[asrini@consign ~]$ cp -i /home/apps/user_defaults/skel/.{bashrc,bash_profile} $HOME/
[asrini@consign ~]$ logout
Connection to consign.pmacs.upenn.edu closed
% ssh consign.pmacs.upenn.edu
[asrini@consign ~]$
NOTE: The last step in the above command set is required - a clean profile is required to ensure the environment variables are all reset correctly.
Once the system defaults have been restored (after logout and subsequent login, into the HPC), the non-Conda related changes can be restored from the $HOME/.bashrc_bak and $HOME/.bash_profile_bak files.
We recommend only enabling/activating Conda environments as needed i.e manually, to avoid issues.
