HPC:Python

* virtualenv             # create own python isolated workspaces http://pypi.python.org/pypi/virtualenv
* virtualenvwrapper      # helpers for virtualenv http://www.doughellmann.com/docs/virtualenvwrapper/index.html#introduction
* pip                    # python package installer
* ipython                # a better python REPL http://ipython.org/
* numpy
* scipy
* matplotlib             # 2D plotting library http://matplotlib.sourceforge.net/index.html
* biopython              # BioPython tools for bioinformatics http://biopython.org/wiki/Main_Page

Python 2.7.5 is also installed on the cluster nodes with the above packages. You can use the `python-2.7.5` environment module

 
echo "module add python-2.7.5" >> ~/.bashrc

Python Virtualenv

We recommend usage of virtualenv to create Python environments in one's own home area on the cluster. As an example, here we want to create a virtualenv with the Python 2.7.5 installation. To do this, please enter the following in a shell:

bsub -Is bash 
module add python-2.7.5
virtualenv $HOME/my_python-2.7.5  --system-site-packages
source $HOME/my_python-2.7.5/bin/activate

The above alters your current PATH to include the Python 2.7.5 executable. It then creates a new virtualenv that includes all of the system installed packages listed above. You will now be able to install your own python packages as needed using pip.

To default to using this environment during batch (bsub) and interactive (bsub -Is bash) sessions, add this to the bottom of your .bashrc file:

if [ $HOSTNAME != "consign.hpc.local" ] && [ $HOSTNAME != "mercury.pmacs.upenn.edu" ]; then
  source $HOME/my_python-2.7.5/bin/activate
fi

Installing Python packages in a virtualenv inside your home area

Here is a session example of the above instructions, including and example installation of a local package using pip

[asrini@consign ~]$ bsub -Is bash
Job <172231> is submitted to default queue <interactive>.
<<Waiting for dispatch ...>>
<<Starting on node064.hpc.local>>
[asrini@node064 ~]$ module add python-2.7.5
[asrini@node064 ~]$
[asrini@node064 ~]$ virtualenv $HOME/my_python-2.7.5  --system-site-packages
New python executable in /home/asrini/my_python-2.7.5/bin/python
Installing Setuptools...................................................................................................................................................................................................................................done.
Installing Pip..........................................................................................................................................................................................................................................................................................................................................done.
[asrini@node064 ~]$ source $HOME/my_python-2.7.5/bin/activate
(my_python-2.7.5)[asrini@node064 ~]$
(my_python-2.7.5)[asrini@node064 ~]$ which python
~/my_python-2.7.5/bin/python
(my_python-2.7.5)[asrini@node064 ~]$ python --version
Python 2.7.5
(my_python-2.7.5)[asrini@node064 ~]$ ipython
WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
Python 2.7.5 (default, Aug 19 2013, 22:59:01)
Type "copyright", "credits" or "license" for more information.

IPython 1.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]:  import numpy

In [2]: exit()
(my_python-2.7.5)[asrini@node064 ~]$
(my_python-2.7.5)[asrini@node064 ~]$ pip install bottle
Downloading/unpacking bottle
  Downloading bottle-0.12.5.tar.gz (69kB): 69kB downloaded
  Running setup.py egg_info for package bottle

Installing collected packages: bottle
  Running setup.py install for bottle
    changing mode of build/scripts-2.7/bottle.py from 664 to 775

    changing mode of /home/asrini/my_python-2.7.5/bin/bottle.py to 775
Successfully installed bottle
Cleaning up...
(my_python-2.7.5)[asrini@node064 ~]$ which bottle.py
~/my_python-2.7.5/bin/bottle.py

Shown above is using virtualenv to create a new custom python environment in your home folder that pulled in numpy/scipy and other system install python libraries . Also shown was installation of a local python package.

HPC:Python

Contents

Python

Python packages available across all PMACS HPC cluster nodes

Python Virtualenv

Installing Python packages in a virtualenv inside your home area

Other pages