Difference between revisions of "HPC:Python"
From HPC wiki
| Line 1: | Line 1: | ||
=== Python === | === Python === | ||
| − | Python 2.6.6 is installed on all cluster nodes, along with | + | Python 2.6.6 is installed on all cluster nodes, along with several other packages. |
| + | |||
| + | ==== Python packages available across all PMACS HPC cluster nodes ==== | ||
<pre> | <pre> | ||
| Line 38: | Line 40: | ||
source $HOME/my_python-2.7.5/bin/activate | source $HOME/my_python-2.7.5/bin/activate | ||
</pre> | </pre> | ||
| + | |||
| + | ==== Installing Python packages in a virtualenv inside your home area ==== | ||
Here is a session example of the above instructions, including and example installation of a local package using <code>pip </code> | Here is a session example of the above instructions, including and example installation of a local package using <code>pip </code> | ||
| + | |||
<pre> | <pre> | ||
[asrini@consign ~]$ bsub -Is bash | [asrini@consign ~]$ bsub -Is bash | ||
| Line 89: | Line 94: | ||
Shown above is using virtualenv to create a new custom python environment in your home folder that pulled in numpy/scipy and other system install python libraries . Also shown was installation of a local python package. | Shown above is using virtualenv to create a new custom python environment in your home folder that pulled in numpy/scipy and other system install python libraries . Also shown was installation of a local python package. | ||
| − | |||
=== Other pages === | === Other pages === | ||
Revision as of 00:52, 1 April 2014
Contents
Python
Python 2.6.6 is installed on all cluster nodes, along with several other packages.
Python packages available across all PMACS HPC cluster nodes
* virtualenv # create own python isolated workspaces http://pypi.python.org/pypi/virtualenv * virtualenvwrapper # helpers for virtualenv http://www.doughellmann.com/docs/virtualenvwrapper/index.html#introduction * pip # python package installer * ipython # a better python REPL http://ipython.org/ * numpy * scipy * matplotlib # 2D plotting library http://matplotlib.sourceforge.net/index.html * biopython # BioPython tools for bioinformatics http://biopython.org/wiki/Main_Page
Python 2.7.5 is also installed on the cluster nodes with the above packages. You can use the `python-2.7.5` environment module
echo "module add python-2.7.5" >> ~/.bashrc
Python Virtualenv
We recommend usage of virtualenv to create their own Python environments on the cluster. As an example, here we want to create a virtualenv with the Python 2.7.5 installation. To do this, please enter the following in a shell:
bsub -Is bash module add python-2.7.5 virtualenv $HOME/my_python-2.7.5 --system-site-packages source $HOME/my_python-2.7.5/bin/activate
The above alters your current PATH to include the Python 2.7.5 executable. It then creates a new virtualenv that includes all of the system installed packages listed above. You will now be able to install your own python packages as needed using pip.
To default to using this environment during qsub and qlogin sessions, add this to the bottom of your .bashrc file:
source $HOME/my_python-2.7.5/bin/activate
Installing Python packages in a virtualenv inside your home area
Here is a session example of the above instructions, including and example installation of a local package using pip
[asrini@consign ~]$ bsub -Is bash
Job <172231> is submitted to default queue <interactive>.
<<Waiting for dispatch ...>>
<<Starting on node064.hpc.local>>
[asrini@node064 ~]$ module add python-2.7.5
[asrini@node064 ~]$
[asrini@node064 ~]$ virtualenv $HOME/my_python-2.7.5 --system-site-packages
New python executable in /home/asrini/my_python-2.7.5/bin/python
Installing Setuptools...................................................................................................................................................................................................................................done.
Installing Pip..........................................................................................................................................................................................................................................................................................................................................done.
[asrini@node064 ~]$ source $HOME/my_python-2.7.5/bin/activate
(my_python-2.7.5)[asrini@node064 ~]$
(my_python-2.7.5)[asrini@node064 ~]$ which python
~/my_python-2.7.5/bin/python
(my_python-2.7.5)[asrini@node064 ~]$ python --version
Python 2.7.5
(my_python-2.7.5)[asrini@node064 ~]$ ipython
WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
Python 2.7.5 (default, Aug 19 2013, 22:59:01)
Type "copyright", "credits" or "license" for more information.
IPython 1.0.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import numpy
In [2]: exit()
(my_python-2.7.5)[asrini@node064 ~]$
(my_python-2.7.5)[asrini@node064 ~]$ pip install bottle
Downloading/unpacking bottle
Downloading bottle-0.12.5.tar.gz (69kB): 69kB downloaded
Running setup.py egg_info for package bottle
Installing collected packages: bottle
Running setup.py install for bottle
changing mode of build/scripts-2.7/bottle.py from 664 to 775
changing mode of /home/asrini/my_python-2.7.5/bin/bottle.py to 775
Successfully installed bottle
Cleaning up...
(my_python-2.7.5)[asrini@node064 ~]$ which bottle.py
~/my_python-2.7.5/bin/bottle.py
Shown above is using virtualenv to create a new custom python environment in your home folder that pulled in numpy/scipy and other system install python libraries . Also shown was installation of a local python package.
