HPC:Conda

This section of the wiki discusses the use of Conda/Anaconda on the HPC.

Conda

Conda is a cross-platform, language-agnostic binary package manager. While using tools like Conda/Anaconda makes it easy for local package management and reusing Python virtual environments, invariably, it also allows most novice (and in some cases experienced) HPC users to inadvertently overwrite their .bashrc and/or .bash_profile files. Thus modifying crucial environment variables such as $PATH etc. We encourage users to tread with caution when using Conda.

Below is an example of a common issue that is caused by local Conda installations:

[asrini@consign ~]$ bsub -Is bash
Job <64070984> is submitted to default queue <interactive>.
<<Waiting for dispatch ...>>
<<Starting on node082.hpc.local>

(base) bash-4.2$ module avail
module avail
bash: module: command not found

The fix for the above to to make a copy of the Conda based .bashrc and .bash_profile files and then use the system default versions of these files and then re-apply only the non-Conda related changes in the now-older .bashrc/.bash_profile files:

[asrini@consign ~]$ mv $HOME/.bashrc $HOME/.bashrc_bak

[asrini@consign ~]$ mv $HOME/.bash_profile $HOME/.bash_profile_bak

[asrini@consign ~]$ cp -i /home/apps/user_defaults/skel/.{bashrc,bash_profile} $HOME/

[asrini@consign ~]$ logout
Connection to consign.pmacs.upenn.edu closed

% ssh consign.pmacs.upenn.edu
[asrini@consign ~]$ 

NOTE: The last step in the above command set is required - a clean profile is required to ensure the environment variables are all reset correctly.

Once the system defaults have been restored (after logout and subsequent login, into the HPC), the non-Conda related changes can be restored from the $HOME/.bashrc_bak and $HOME/.bash_profile_bak files.

We recommend only enabling/activating Conda environments as needed i.e manually, to avoid issues.

Other Pages

---

• HPC Main Page
• HPC FAQ
• Connecting to the PMACS cluster
• User Guide
• Available Software
• PMACS Archive System