Difference between revisions of "HPC:Python"

From HPC wiki
 
(7 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
=== Python ===
 
=== Python ===
Python 2.6.6 is installed on all cluster nodes, along with several other packages.
+
Currently, there are 4 different versions of Python installed on the PMACS cluster. The system default version is Python v2.6.6. The other versions of Python installed are: v2.7.5, v2.7.9 & v3.4.2
  
 
==== Python packages available across all PMACS HPC cluster nodes ====
 
==== Python packages available across all PMACS HPC cluster nodes ====
Line 15: Line 15:
 
</pre>
 
</pre>
  
Python 2.7.5 is also installed on the cluster nodes with the above packages. You can use the `python-2.7.5` environment module
+
==== Python 2.7.5 ====
 +
To use a different version of Python, use the appropriate Environment Module. For Python v2.7.5, for example, you can use the `python-2.7.5` environment module
 +
 
 +
<pre>
 +
 
 +
[asrini@node062 ~]$ python --version
 +
Python 2.6.6
 +
 
 +
[asrini@node062 ~]$ which python
 +
/usr/bin/python
 +
 
 +
[asrini@node062 ~]$ module load python-2.7.5
 +
 
 +
[asrini@node062 ~]$ which python
 +
/opt/software/python/2.7.5/bin/python
 +
 
 +
[asrini@node062 ~]$ python --version
 +
Python 2.7.5
 +
</pre>
  
 
<pre>  
 
<pre>  
 
echo "module add python-2.7.5" >> ~/.bashrc
 
echo "module add python-2.7.5" >> ~/.bashrc
 +
</pre>
 +
 +
 +
==== Python 2.7.9 ====
 +
To use a different version of Python, use the appropriate Environment Module. For Python v2.7.9, for example, you can use the `python-2.7.9` environment module
 +
 +
<pre>
 +
[asrini@node062 ~]$ module load python-2.7.9
 +
 +
[asrini@node062 ~]$ python --version
 +
Python 2.7.9
 +
</pre>
 +
 +
==== Python 3.4.2 ====
 +
To use a different version of Python, use the appropriate Environment Module. For Python v3.4.2, for example, you can use the `python-3.4.2` environment module
 +
 +
<pre>
 +
[asrini@node062 ~]$ module load python-3.4.2
 +
 +
[asrini@node062 ~]$ python --version
 +
Python 3.4.2
 +
</pre>
 +
 +
'''NOTE: module naming conventions have changed. Run "module avail" first to see what the new module names are, before attempting to load the module.'''
 +
 +
==== Python 3.6.3 ====
 +
To use a different version of Python, use the appropriate Environment Module. For Python v3.6.3, for example, you can use the `python/3.6.3` environment module
 +
 +
<pre>
 +
[asrini@node156 ~]$ module load python/3.6.3
 +
 +
[asrini@node156 ~]$ python --version
 +
Python 3.6.3
 
</pre>
 
</pre>
  
 
==== Python Virtualenv ====
 
==== Python Virtualenv ====
  
We recommend usage of <code>virtualenv</code> to create their own Python environments on the cluster. As an example, here we want to create a virtualenv with the Python 2.7.5 installation. To do this, please enter the following in a shell:
+
We recommend usage of <code>virtualenv</code> to create Python environments in one's own home area on the cluster. As an example, here we want to create a virtualenv with the Python 2.7.5 installation. To do this, please enter the following in a shell:
  
 
<pre>
 
<pre>
Line 35: Line 86:
  
  
To default to using this environment during batch (bsub) and interactive (bsub -Is bash) sessions, add this to the bottom of your .bashrc file:
+
To default to using this environment during an interactive (bsub -Is bash) session, add this to the bottom of your .bashrc file:
  
 
<pre>
 
<pre>
Line 41: Line 92:
 
   source $HOME/my_python-2.7.5/bin/activate
 
   source $HOME/my_python-2.7.5/bin/activate
 
fi
 
fi
 +
</pre>
 +
 +
To include this in a batch (bsub) job add the following line to the top of your batch submission script:
 +
<pre>
 +
  source $HOME/my_python-2.7.5/bin/activate
 
</pre>
 
</pre>
  
Line 97: Line 153:
 
Shown above is using virtualenv to create a new custom python environment in your home folder that pulled in numpy/scipy and other system install python libraries . Also shown was installation of a local python package.
 
Shown above is using virtualenv to create a new custom python environment in your home folder that pulled in numpy/scipy and other system install python libraries . Also shown was installation of a local python package.
  
=== Other pages ===
+
==== Running Python scripts non-interactively ====
 +
 
 +
Shown below is a sample Python script:
 +
<pre>
 +
$ cat simple.py
 +
#!/usr/bin/env python
 +
print "Hello!"
 +
</pre>
 +
 
 +
The above script can be run on the PMACS HPC cluster:
 +
 
 +
<pre>
 +
$ bsub -e simple.e -o simple.o python simple.py
 +
</pre>
 +
 
 +
 
 +
=== Other Pages ===
 +
----
 +
<div class="mw-collapsible mw-collapsed">
 +
*[[HPC:FAQ|HPC FAQ ]]
 +
*[[HPC:Login|Connecting to the PMACS cluster]]
 +
*[[HPC:User_Guide|User Guide]]
 
*[[HPC:Software|Available Software]]
 
*[[HPC:Software|Available Software]]
*[[HPC:User_Guide|User Guide]]
+
*[[HPC:Archive System|PMACS Archive System]]
*[[HPC:Main_Page|HPC Main Page]]
+
</div>
*[[HPC:Login|Connecting to the PMACS cluster]]
 

Latest revision as of 18:06, 7 December 2020

Python

Currently, there are 4 different versions of Python installed on the PMACS cluster. The system default version is Python v2.6.6. The other versions of Python installed are: v2.7.5, v2.7.9 & v3.4.2

Python packages available across all PMACS HPC cluster nodes

* virtualenv             # create own python isolated workspaces http://pypi.python.org/pypi/virtualenv
* virtualenvwrapper      # helpers for virtualenv http://www.doughellmann.com/docs/virtualenvwrapper/index.html#introduction
* pip                    # python package installer
* ipython                # a better python REPL http://ipython.org/
* numpy
* scipy
* matplotlib             # 2D plotting library http://matplotlib.sourceforge.net/index.html
* biopython              # BioPython tools for bioinformatics http://biopython.org/wiki/Main_Page

Python 2.7.5

To use a different version of Python, use the appropriate Environment Module. For Python v2.7.5, for example, you can use the `python-2.7.5` environment module


[asrini@node062 ~]$ python --version
Python 2.6.6

[asrini@node062 ~]$ which python
/usr/bin/python

[asrini@node062 ~]$ module load python-2.7.5

[asrini@node062 ~]$ which python
/opt/software/python/2.7.5/bin/python

[asrini@node062 ~]$ python --version
Python 2.7.5
 
echo "module add python-2.7.5" >> ~/.bashrc


Python 2.7.9

To use a different version of Python, use the appropriate Environment Module. For Python v2.7.9, for example, you can use the `python-2.7.9` environment module

[asrini@node062 ~]$ module load python-2.7.9

[asrini@node062 ~]$ python --version
Python 2.7.9

Python 3.4.2

To use a different version of Python, use the appropriate Environment Module. For Python v3.4.2, for example, you can use the `python-3.4.2` environment module

[asrini@node062 ~]$ module load python-3.4.2

[asrini@node062 ~]$ python --version
Python 3.4.2

NOTE: module naming conventions have changed. Run "module avail" first to see what the new module names are, before attempting to load the module.

Python 3.6.3

To use a different version of Python, use the appropriate Environment Module. For Python v3.6.3, for example, you can use the `python/3.6.3` environment module

[asrini@node156 ~]$ module load python/3.6.3 

[asrini@node156 ~]$ python --version
Python 3.6.3

Python Virtualenv

We recommend usage of virtualenv to create Python environments in one's own home area on the cluster. As an example, here we want to create a virtualenv with the Python 2.7.5 installation. To do this, please enter the following in a shell:

bsub -Is bash 
module add python-2.7.5
virtualenv $HOME/my_python-2.7.5  --system-site-packages
source $HOME/my_python-2.7.5/bin/activate

The above alters your current PATH to include the Python 2.7.5 executable. It then creates a new virtualenv that includes all of the system installed packages listed above. You will now be able to install your own python packages as needed using pip.


To default to using this environment during an interactive (bsub -Is bash) session, add this to the bottom of your .bashrc file:

if [ $HOSTNAME != "consign.hpc.local" ] && [ $HOSTNAME != "mercury.pmacs.upenn.edu" ]; then
  source $HOME/my_python-2.7.5/bin/activate
fi

To include this in a batch (bsub) job add the following line to the top of your batch submission script:

  source $HOME/my_python-2.7.5/bin/activate

Installing Python packages in a virtualenv inside your home area

Here is a session example of the above instructions, including and example installation of a local package using pip

[asrini@consign ~]$ bsub -Is bash
Job <172231> is submitted to default queue <interactive>.
<<Waiting for dispatch ...>>
<<Starting on node064.hpc.local>>
[asrini@node064 ~]$ module add python-2.7.5
[asrini@node064 ~]$
[asrini@node064 ~]$ virtualenv $HOME/my_python-2.7.5  --system-site-packages
New python executable in /home/asrini/my_python-2.7.5/bin/python
Installing Setuptools...................................................................................................................................................................................................................................done.
Installing Pip..........................................................................................................................................................................................................................................................................................................................................done.
[asrini@node064 ~]$ source $HOME/my_python-2.7.5/bin/activate
(my_python-2.7.5)[asrini@node064 ~]$
(my_python-2.7.5)[asrini@node064 ~]$ which python
~/my_python-2.7.5/bin/python
(my_python-2.7.5)[asrini@node064 ~]$ python --version
Python 2.7.5
(my_python-2.7.5)[asrini@node064 ~]$ ipython
WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
Python 2.7.5 (default, Aug 19 2013, 22:59:01)
Type "copyright", "credits" or "license" for more information.

IPython 1.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]:  import numpy

In [2]: exit()
(my_python-2.7.5)[asrini@node064 ~]$
(my_python-2.7.5)[asrini@node064 ~]$ pip install bottle
Downloading/unpacking bottle
  Downloading bottle-0.12.5.tar.gz (69kB): 69kB downloaded
  Running setup.py egg_info for package bottle

Installing collected packages: bottle
  Running setup.py install for bottle
    changing mode of build/scripts-2.7/bottle.py from 664 to 775

    changing mode of /home/asrini/my_python-2.7.5/bin/bottle.py to 775
Successfully installed bottle
Cleaning up...
(my_python-2.7.5)[asrini@node064 ~]$ which bottle.py
~/my_python-2.7.5/bin/bottle.py

Shown above is using virtualenv to create a new custom python environment in your home folder that pulled in numpy/scipy and other system install python libraries . Also shown was installation of a local python package.

Running Python scripts non-interactively

Shown below is a sample Python script:

$ cat simple.py
#!/usr/bin/env python
print "Hello!"

The above script can be run on the PMACS HPC cluster:

$ bsub -e simple.e -o simple.o python simple.py


Other Pages