Difference between revisions of "HPC:Python"

Revision as of 18:05, 7 December 2020

* virtualenv             # create own python isolated workspaces http://pypi.python.org/pypi/virtualenv
* virtualenvwrapper      # helpers for virtualenv http://www.doughellmann.com/docs/virtualenvwrapper/index.html#introduction
* pip                    # python package installer
* ipython                # a better python REPL http://ipython.org/
* numpy
* scipy
* matplotlib             # 2D plotting library http://matplotlib.sourceforge.net/index.html
* biopython              # BioPython tools for bioinformatics http://biopython.org/wiki/Main_Page

Python 2.7.5

To use a different version of Python, use the appropriate Environment Module. For Python v2.7.5, for example, you can use the `python-2.7.5` environment module


[asrini@node062 ~]$ python --version
Python 2.6.6

[asrini@node062 ~]$ which python
/usr/bin/python

[asrini@node062 ~]$ module load python-2.7.5

[asrini@node062 ~]$ which python
/opt/software/python/2.7.5/bin/python

[asrini@node062 ~]$ python --version
Python 2.7.5

 
echo "module add python-2.7.5" >> ~/.bashrc

Python 2.7.9

To use a different version of Python, use the appropriate Environment Module. For Python v2.7.9, for example, you can use the `python-2.7.9` environment module

[asrini@node062 ~]$ module load python-2.7.9

[asrini@node062 ~]$ python --version
Python 2.7.9

Python 3.4.2

To use a different version of Python, use the appropriate Environment Module. For Python v3.4.2, for example, you can use the `python-3.4.2` environment module

[asrini@node062 ~]$ module load python-3.4.2

[asrini@node062 ~]$ python --version
Python 3.4.2

NOTE: module naming conventions have changed. run "module avail" first to see what the new module names are, before attempting to load the module.

Python 3.6.3

To use a different version of Python, use the appropriate Environment Module. For Python v3.6.3, for example, you can use the `python/3.6.3` environment module

[asrini@node156 ~]$ module load python/3.6.3 

[asrini@node156 ~]$ python --version
Python 3.6.3

Python Virtualenv

We recommend usage of virtualenv to create Python environments in one's own home area on the cluster. As an example, here we want to create a virtualenv with the Python 2.7.5 installation. To do this, please enter the following in a shell:

bsub -Is bash 
module add python-2.7.5
virtualenv $HOME/my_python-2.7.5  --system-site-packages
source $HOME/my_python-2.7.5/bin/activate

The above alters your current PATH to include the Python 2.7.5 executable. It then creates a new virtualenv that includes all of the system installed packages listed above. You will now be able to install your own python packages as needed using pip.

To default to using this environment during an interactive (bsub -Is bash) session, add this to the bottom of your .bashrc file:

if [ $HOSTNAME != "consign.hpc.local" ] && [ $HOSTNAME != "mercury.pmacs.upenn.edu" ]; then
  source $HOME/my_python-2.7.5/bin/activate
fi

To include this in a batch (bsub) job add the following line to the top of your batch submission script:

  source $HOME/my_python-2.7.5/bin/activate

Installing Python packages in a virtualenv inside your home area

Here is a session example of the above instructions, including and example installation of a local package using pip

[asrini@consign ~]$ bsub -Is bash
Job <172231> is submitted to default queue <interactive>.
<<Waiting for dispatch ...>>
<<Starting on node064.hpc.local>>
[asrini@node064 ~]$ module add python-2.7.5
[asrini@node064 ~]$
[asrini@node064 ~]$ virtualenv $HOME/my_python-2.7.5  --system-site-packages
New python executable in /home/asrini/my_python-2.7.5/bin/python
Installing Setuptools...................................................................................................................................................................................................................................done.
Installing Pip..........................................................................................................................................................................................................................................................................................................................................done.
[asrini@node064 ~]$ source $HOME/my_python-2.7.5/bin/activate
(my_python-2.7.5)[asrini@node064 ~]$
(my_python-2.7.5)[asrini@node064 ~]$ which python
~/my_python-2.7.5/bin/python
(my_python-2.7.5)[asrini@node064 ~]$ python --version
Python 2.7.5
(my_python-2.7.5)[asrini@node064 ~]$ ipython
WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
Python 2.7.5 (default, Aug 19 2013, 22:59:01)
Type "copyright", "credits" or "license" for more information.

IPython 1.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]:  import numpy

In [2]: exit()
(my_python-2.7.5)[asrini@node064 ~]$
(my_python-2.7.5)[asrini@node064 ~]$ pip install bottle
Downloading/unpacking bottle
  Downloading bottle-0.12.5.tar.gz (69kB): 69kB downloaded
  Running setup.py egg_info for package bottle

Installing collected packages: bottle
  Running setup.py install for bottle
    changing mode of build/scripts-2.7/bottle.py from 664 to 775

    changing mode of /home/asrini/my_python-2.7.5/bin/bottle.py to 775
Successfully installed bottle
Cleaning up...
(my_python-2.7.5)[asrini@node064 ~]$ which bottle.py
~/my_python-2.7.5/bin/bottle.py

Shown above is using virtualenv to create a new custom python environment in your home folder that pulled in numpy/scipy and other system install python libraries . Also shown was installation of a local python package.

Running Python scripts non-interactively

Shown below is a sample Python script:

$ cat simple.py
#!/usr/bin/env python
print "Hello!"

The above script can be run on the PMACS HPC cluster:

$ bsub -e simple.e -o simple.o python simple.py

Difference between revisions of "HPC:Python"

Revision as of 18:05, 7 December 2020

Contents

Python

Python packages available across all PMACS HPC cluster nodes