Difference between revisions of "HPC:Environment Modules"
From HPC wiki
(Created page with "=== Environment Modules === The PMACS cluster utilizes modules to enable users to alter their environment. These modules alter your execution path (<code>$PATH</code>) and add...") |
|||
| Line 18: | Line 18: | ||
bowtie2-2.1.0 manpath openmpi-1.5.4-x86_64 ruby-1.8.7-p374 | bowtie2-2.1.0 manpath openmpi-1.5.4-x86_64 ruby-1.8.7-p374 | ||
devtoolset-2 module-cvs perl5lib ruby-1.9.3-p448 | devtoolset-2 module-cvs perl5lib ruby-1.9.3-p448 | ||
| + | </pre> | ||
| + | |||
| + | |||
| + | The module names should be pretty self-explainatory, but some are not. To see information about a module | ||
| + | you can issue a <tt>module show [module name]</tt>: | ||
| + | |||
| + | <pre> | ||
| + | [asrini@node063 ~]$ module show null | ||
| + | ------------------------------------------------------------------- | ||
| + | /usr/share/Modules/modulefiles/null: | ||
| + | |||
| + | module-whatis does absolutely nothing | ||
| + | ------------------------------------------------------------------- | ||
| + | |||
| + | [asrini@node063 ~]$ module show r-libs-user | ||
| + | ------------------------------------------------------------------- | ||
| + | /usr/share/Modules/modulefiles/r-libs-user: | ||
| + | |||
| + | module-whatis Sets R_LIBS_USER=$HOME/R/library | ||
| + | setenv R_LIBS_USER ~/R/library | ||
| + | ------------------------------------------------------------------- | ||
| + | |||
| + | [asrini@node063 ~]$ module show devtoolset-2 | ||
| + | ------------------------------------------------------------------- | ||
| + | /usr/share/Modules/modulefiles/devtoolset-2: | ||
| + | |||
| + | module-whatis Devtoolset-2 packages include the newer versions of gcc | ||
| + | prepend-path PATH /opt/rh/devtoolset-2/root/usr/bin | ||
| + | prepend-path MANPATH /opt/rh/devtoolset-2/root/usr/share/man | ||
| + | prepend-path INFOPATH /opt/rh/devtoolset-2/root/usr/share/info | ||
| + | ------------------------------------------------------------------- | ||
| + | |||
</pre> | </pre> | ||
Example use of modules: | Example use of modules: | ||
| − | <pre> | + | <pre>[asrini@node063 ~]$ python -V |
| − | [asrini@node063 ~]$ python -V | ||
Python 2.6.6 | Python 2.6.6 | ||
| Line 45: | Line 76: | ||
[asrini@node063 ~]$ which python | [asrini@node063 ~]$ which python | ||
/usr/bin/python | /usr/bin/python | ||
| − | |||
</pre> | </pre> | ||
Modules may also be included in your job scripts and submitted as a batch job. | Modules may also be included in your job scripts and submitted as a batch job. | ||
| + | |||
| + | ==== Using Modules at Login ==== | ||
| + | |||
| + | In order to have modules automatically load into your environment, you would add the module commands | ||
| + | to your <tt>$HOME/.bashrc</tt> file. Note that modules are not available on the PMACS head node, hence, you'll need to ensure that your login script attempts to load a module only if you are on a compute node: | ||
| + | |||
| + | <pre> | ||
| + | [asrini@consign ~]$ more .bashrc | ||
| + | # .bashrc | ||
| + | |||
| + | # Source global definitions | ||
| + | if [ -f /etc/bashrc ]; then | ||
| + | . /etc/bashrc | ||
| + | fi | ||
| + | # | ||
| + | # | ||
| + | # Modules to load | ||
| + | if [ $HOSTNAME != "consign.hpc.local" ] && [ $HOSTNAME != "mercury.pmacs.upenn.edu" ]; then | ||
| + | module load python-2.7.5 | ||
| + | fi | ||
| + | |||
| + | # more stuff below ..... | ||
| + | |||
| + | [asrini@consign ~]$ which python | ||
| + | /usr/bin/python | ||
| + | [asrini@consign ~]$ bsub -Is bash | ||
| + | Job <172129> is submitted to default queue <interactive>. | ||
| + | <<Waiting for dispatch ...>> | ||
| + | <<Starting on node063.hpc.local>> | ||
| + | [asrini@node063 ~]$ which python | ||
| + | /opt/software/python/python-2.7.5/bin/python | ||
| + | </pre> | ||
Revision as of 17:33, 19 March 2014
Environment Modules
The PMACS cluster utilizes modules to enable users to alter their environment. These modules alter your execution path ($PATH) and add or modify the user's environment variables. User loadable modules are available if the system default packages don't meet your requirements. Environment modules are used in place of hard-coding paths in your $HOME/.bashrc file.
To know what modules are available, you'll need to run the "module avail" command from an interactive session:
[asrini@consign ~]$ bsub -Is bash
Job <9990024> is submitted to default queue <interactive>.
<<Waiting for dispatch ...>>
<<Starting on node063.hpc.local>>
[asrini@node063 ~]$ module avail
------------------------------------------------------------------- /usr/share/Modules/modulefiles -------------------------------------------------------------------
NAMD-2.9-Linux-x86_64-multicore dot module-info picard-1.96 rum-2.0.5_05
STAR-2.3.0e java-sdk-1.6.0 modules pkg-config-path samtools-0.1.19
STAR-hg19 java-sdk-1.7.0 mpich2-x86_64 python-2.7.5 use.own
STAR-mm9 ld-library-path null r-libs-user
bowtie2-2.1.0 manpath openmpi-1.5.4-x86_64 ruby-1.8.7-p374
devtoolset-2 module-cvs perl5lib ruby-1.9.3-p448
The module names should be pretty self-explainatory, but some are not. To see information about a module
you can issue a module show [module name]:
[asrini@node063 ~]$ module show null ------------------------------------------------------------------- /usr/share/Modules/modulefiles/null: module-whatis does absolutely nothing ------------------------------------------------------------------- [asrini@node063 ~]$ module show r-libs-user ------------------------------------------------------------------- /usr/share/Modules/modulefiles/r-libs-user: module-whatis Sets R_LIBS_USER=$HOME/R/library setenv R_LIBS_USER ~/R/library ------------------------------------------------------------------- [asrini@node063 ~]$ module show devtoolset-2 ------------------------------------------------------------------- /usr/share/Modules/modulefiles/devtoolset-2: module-whatis Devtoolset-2 packages include the newer versions of gcc prepend-path PATH /opt/rh/devtoolset-2/root/usr/bin prepend-path MANPATH /opt/rh/devtoolset-2/root/usr/share/man prepend-path INFOPATH /opt/rh/devtoolset-2/root/usr/share/info -------------------------------------------------------------------
Example use of modules:
[asrini@node063 ~]$ python -V Python 2.6.6 [asrini@node063 ~]$ which python /usr/bin/python [asrini@node063 ~]$ module load python-2.7.5 [asrini@node063 ~]$ python -V Python 2.7.5 [asrini@node063 ~]$ which python /opt/software/python/python-2.7.5/bin/python
After running the above commands, you will be able to use python v2.7.5 till you exit out of the interactive session or till you unload the module:
[asrini@node063 ~]$ module unload python-2.7.5 [asrini@node063 ~]$ which python /usr/bin/python
Modules may also be included in your job scripts and submitted as a batch job.
Using Modules at Login
In order to have modules automatically load into your environment, you would add the module commands to your $HOME/.bashrc file. Note that modules are not available on the PMACS head node, hence, you'll need to ensure that your login script attempts to load a module only if you are on a compute node:
[asrini@consign ~]$ more .bashrc # .bashrc # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi # # # Modules to load if [ $HOSTNAME != "consign.hpc.local" ] && [ $HOSTNAME != "mercury.pmacs.upenn.edu" ]; then module load python-2.7.5 fi # more stuff below ..... [asrini@consign ~]$ which python /usr/bin/python [asrini@consign ~]$ bsub -Is bash Job <172129> is submitted to default queue <interactive>. <<Waiting for dispatch ...>> <<Starting on node063.hpc.local>> [asrini@node063 ~]$ which python /opt/software/python/python-2.7.5/bin/python
