HPC:Jupyter
Contents
Jupyter Notebook
Currently, the HPC is capable of hosting a Jupyter Notebook workspace in an interactive session. This limits the Jupyter Notebook to an 8 hour session. Please follow these steps to set up your Jupyter Notebook workspace.
The initial setup for Jupyter is a little tricky, but once completed is easy to get up and running again.
This guide will assume that you are using Windows 10 MobaXTerm as your terminal emulator in order to set up SSH tunneling. However, this can also be done via other tools (Powershell, PuTTy, et cetera) but those tools will not be covered in this guide. If you have questions about setting up Jupyter with other tools, please contact the HPC team (or come to office hours!)
First, install the Jupyter client.
[obki@consign ~]$ bsub -Is bash << Waiting for dispatch...>> <<Starting on node157.hpc.local>> [obki@node157 ~]$ module load python/3.6.3 gcc/8.3.1 [obki@node157 ~]$ pip3 install jupyter --user
At this point Jupyter should be installed via pip3. To make sure that it installed properly, run the following command.
[obki@node157 ~]$ ls $HOME/.local/bin/jupyter /home/obki/.local/bin/jupyter
This regular file should be the output of the ls command. Next, we can start the the Jupyter host from an interactive session using the following command.
[obki@node157 ~]$ $HOME/.local/bin/jupyter notebook --ip `hostname`
[I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki
[I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at:
[I 16:13:48.696 NotebookApp] http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.696 NotebookApp] or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 16:13:48.707 NotebookApp] No web browser found: could not locate runnable browser.
[C 16:13:48.707 NotebookApp]
To access the notebook, open this file in a browser:
file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html
Or copy and paste one of these URLs:
http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
Next, we have to set up an SSH tunnel to connect to the webpage.
Setup SSH Tunnel on macOS/Linux Clients
If you have a Windows laptop/desktop, see the section below.
To setup an SSH tunnel to the Jupyter notebook session that was previously created, on a macOS or Linux client, use the following steps.
Step 1: Create the tunnel using the following command:
ssh -L 8888:node157:8888 <your_user_name>@consign.pmacs.upenn.edu
NOTE 1: Since the remote interactive session was assigned to "node157" the above SSH forwarding command must be set to same node name. Change the node name as needed.
Step 2: Click the localhost specific link generated in the output of the "jupyter notebook --ip `hostname`" command (the link that starts with "http:// 127.0.0.1").
In the above example, that would be:
http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
NOTE 2: The output of the "jupyter notebook --ip `hostname`" command produces a unique URL with a unique token. Ensure that you are clicking the link in your specific command output. DO NOT bookmark/save this link for future use.
Setup SSH Tunnel on Windows Clients
First, open the "Tunnel" feature on MobaXterm
Create a new tunnel
We'll be using local port forwarding.
Note: The <remote server> field is dependent on what node your interactive session (and Jupyter) is running. For me this is Node157 but may be different for you. Please adjust your input according to the node that you are running Jupyter on
These are the inputs you should use for each input field.
Save your tunnel. Finally, start your tunnel by hitting the "Play" button, or simply by clicking "Start all tunnels"
Congratulations! You've now set up a Jupyter Notebook. Go to a webpage and type localhost:8888 into the URL to access the Jupyter login page. You'll need a token to login, which you can find from the output of the last command you entered into your interactive session.
[I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki
[I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at:
[I 16:13:48.696 NotebookApp] http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.696 NotebookApp] or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 16:13:48.707 NotebookApp] No web browser found: could not locate runnable browser.
[C 16:13:48.707 NotebookApp]
To access the notebook, open this file in a browser:
file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html
Or copy and paste one of these URLs:
http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
So for in my case, my token is fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
Notes
Keep in mind that the Jupyter Notebook only lasts as long as an interactive session does, so 8 hours. To run Jupyter again in the future, simply run $HOME/.local/bin/jupyter notebook --ip `hostname` from an interactive session.
Always remember to edit your SSH tunnel to match the node from which Jupyter is running from.
