Difference between revisions of "HPC:Jupyter"

From HPC wiki
 
(19 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
===Jupyter Notebook===
 
===Jupyter Notebook===
Currently, the HPC is capable of hosting a Jupyter Notebook workspace in an interactive session. Currently, this limits the Jupyter Notebook to an 8 hour session. Please follow these steps to set up your Jupyter Notebook workspace.
+
Currently, the HPC is capable of hosting a Jupyter Notebook workspace in an interactive session. This limits the Jupyter Notebook to an 8 hour session. Please follow these steps to set up your Jupyter Notebook workspace.
  
 
The initial setup for Jupyter is a little tricky, but once completed is easy to get up and running again.  
 
The initial setup for Jupyter is a little tricky, but once completed is easy to get up and running again.  
  
This guide will assume that you are using MobaXTerm as your terminal emulator in order to set up SSH tunneling. However, this can also be done via other tools (Powershell, PuTTy, et cetera) but those tools will not be covered in this guide. If you have questions about setting up Jupyter with other tools, please contact the HPC team (or come to office hours!)
+
This guide will assume that you are using Windows 10 MobaXTerm as your terminal emulator in order to set up SSH tunneling. However, this can also be done via other tools (Powershell, PuTTy, et cetera) but those tools will not be covered in this guide. If you have questions about setting up Jupyter with other tools, please contact the HPC team (or come to office hours!)
  
 
First, install the Jupyter client.
 
First, install the Jupyter client.
Line 10: Line 10:
 
[obki@consign ~]$ bsub -Is bash
 
[obki@consign ~]$ bsub -Is bash
 
<< Waiting for dispatch...>>
 
<< Waiting for dispatch...>>
<<Starting on node059.hpc.local>>
+
<<Starting on node157.hpc.local>>
[obki@node059 ~]$ module load python/3.6.3
+
[obki@node157 ~]$ module load python/3.6.3 gcc/8.3.1
[obki@node059 ~]$ pip3 install jupyter --user
+
[obki@node157 ~]$ pip3 install jupyter --user
 
</pre>
 
</pre>
 
At this point Jupyter should be installed via pip3. To make sure that it installed properly, run the following command.
 
At this point Jupyter should be installed via pip3. To make sure that it installed properly, run the following command.
 
<pre>
 
<pre>
[obki@node059 ~]$ ls $HOME/.local/bin/jupyter
+
[obki@node157 ~]$ ls $HOME/.local/bin/jupyter
 
/home/obki/.local/bin/jupyter
 
/home/obki/.local/bin/jupyter
 
</pre>  
 
</pre>  
This regular file should be the output of the <code>ls</code> command. Next, we can start the the Jupyter host from an interactive session
+
This regular file should be the output of the <code>ls</code> command. Next, we can start the the Jupyter host from an interactive session using the following command.
 
<pre>
 
<pre>
[obki@node059 ~]$ $HOME/.local/bin/jupyter notebook --ip `hostname`
+
[obki@node157 ~]$ $HOME/.local/bin/jupyter notebook --ip `hostname`
 +
 
 
[I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki
 
[I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki
 
[I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at:
 
[I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at:
[I 16:13:48.696 NotebookApp] http://node059.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
+
[I 16:13:48.696 NotebookApp] http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 
[I 16:13:48.696 NotebookApp]  or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 
[I 16:13:48.696 NotebookApp]  or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 
[I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
 
[I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Line 33: Line 34:
 
         file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html
 
         file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html
 
     Or copy and paste one of these URLs:
 
     Or copy and paste one of these URLs:
         http://node059.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
+
         http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 
     or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 
     or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 
</pre>
 
</pre>
  
However, you still cannot bring up the Jupyter Notebook webpage without a few more steps. Firstly, you need to connect to the PMACS VPN.
+
Next, we have to set up an SSH tunnel to connect to the webpage.
Refer to the <strong>[[HPC:Login#More_VPN_Info|login guide]]</strong> on how to install Forticlient VPN and connect to the PMACS VPN
+
 
 +
====Setup SSH Tunnel on macOS/Linux Clients====
 +
If you have a Windows laptop/desktop, see the section [[HPC:Jupyter#Setup_SSH_Tunnel_on_Windows_Clients|below]].
 +
 
 +
To setup an SSH tunnel to the Jupyter notebook session that was previously created, '''on a macOS or Linux''' client, use the following steps.
 +
 
 +
Step 1: Create the tunnel using the following command:
 +
 
 +
<pre>
 +
ssh -L 8888:node157:8888 <your_user_name>@consign.pmacs.upenn.edu
 +
</pre>
 +
 
 +
'''NOTE 1:''' Since the remote interactive session was assigned to "node157" the above SSH forwarding command must be set to same node name. '''Change the node name as needed'''.
  
Once you have connected to the VPN, you will need to set up an SSH tunnel. MobaXTerm makes this process relatively simple with a nice GUI.  
+
Step 2: '''Click the localhost specific link''' generated in the output of the "jupyter notebook --ip `hostname`" command (the link that starts with "http:// 127.0.0.1").
 +
 
 +
In the above example, that would be:
 +
 
 +
<pre>
 +
http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 +
</pre>
 +
 
 +
'''NOTE 2:''' The output of the "jupyter notebook --ip `hostname`" command produces a '''unique''' URL with a '''unique''' token. Ensure that you are clicking the link in your specific command output. '''DO NOT''' bookmark/save this link for future use.
 +
 
 +
====Setup SSH Tunnel on Windows Clients====
  
 
First, open the "Tunnel" feature on MobaXterm
 
First, open the "Tunnel" feature on MobaXterm
[[image:Tunneledit.png|center|600px]]
+
<br />
 +
[[image:TunnelEdit.png|center|600px]]
 +
<br />
 
Create a new tunnel
 
Create a new tunnel
 +
<br />
 
[[image:Tunnel1.PNG|center|600px]]
 
[[image:Tunnel1.PNG|center|600px]]
 +
<br />
 
We'll be using local port forwarding.  
 
We'll be using local port forwarding.  
  
<strong>Note: The <code><remote server></code> field is dependant on what node your interactive session (and Jupyter) is running. For me this is <code>Node059</code> but may be different for you. Please adjust your input according to the node that you are running Jupyter on</strong>
+
<strong>Note: The <code><remote server></code> field is dependent on what node your interactive session (and Jupyter) is running. For me this is <code>Node157</code> but may be different for you. Please adjust your input according to the node that you are running Jupyter on</strong>
  
 
These are the inputs you should use for each input field.  
 
These are the inputs you should use for each input field.  
 +
<br />
 
[[image:Tunnel2.png|center|600px]]  
 
[[image:Tunnel2.png|center|600px]]  
 +
<br />
 
Save your tunnel. Finally, start your tunnel by hitting the "Play" button, or simply by clicking "Start all tunnels"  
 
Save your tunnel. Finally, start your tunnel by hitting the "Play" button, or simply by clicking "Start all tunnels"  
[[image:Tunnel2.png|center|600px]]
+
<br />
 +
[[image:Tunnel3.PNG|center|600px]]
 +
<br />
 +
Congratulations! You've now set up a Jupyter Notebook. Go to a webpage and type <code>localhost:8888</code> into the URL to access the Jupyter login page. You'll need a token to login, which you can find from the output of the last command you entered into your interactive session.
 +
<pre>
 +
[I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki
 +
[I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at:
 +
[I 16:13:48.696 NotebookApp] http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 +
[I 16:13:48.696 NotebookApp]  or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 +
[I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
 +
[W 16:13:48.707 NotebookApp] No web browser found: could not locate runnable browser.
 +
[C 16:13:48.707 NotebookApp]
 +
 
 +
    To access the notebook, open this file in a browser:
 +
        file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html
 +
    Or copy and paste one of these URLs:
 +
        http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 +
    or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
 +
</pre>
 +
 
 +
So for in my case, my token is <code>fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff</code>
 +
===='''Notes'''====
 +
Keep in mind that the Jupyter Notebook only lasts as long as an interactive session does, so 8 hours. To run Jupyter again in the future, simply run <code>$HOME/.local/bin/jupyter notebook --ip `hostname`</code> from an interactive session.
 +
 
 +
Always remember to edit your SSH tunnel to match the node from which Jupyter is running from.
 +
 
 +
=== Other Pages ===
 +
----
 +
<div class="mw-collapsible mw-collapsed">
 +
*[[HPC:Main_Page|HPC Main Page]]
 +
*[[HPC:FAQ|HPC FAQ ]]
 +
*[[HPC:Login|Connecting to the PMACS cluster]]
 +
*[[HPC:Software|Available Software]]
 +
*[[HPC:Archive System|PMACS Archive System]]
 +
</div>

Latest revision as of 19:37, 21 November 2023

Jupyter Notebook

Currently, the HPC is capable of hosting a Jupyter Notebook workspace in an interactive session. This limits the Jupyter Notebook to an 8 hour session. Please follow these steps to set up your Jupyter Notebook workspace.

The initial setup for Jupyter is a little tricky, but once completed is easy to get up and running again.

This guide will assume that you are using Windows 10 MobaXTerm as your terminal emulator in order to set up SSH tunneling. However, this can also be done via other tools (Powershell, PuTTy, et cetera) but those tools will not be covered in this guide. If you have questions about setting up Jupyter with other tools, please contact the HPC team (or come to office hours!)

First, install the Jupyter client.

[obki@consign ~]$ bsub -Is bash
<< Waiting for dispatch...>>
<<Starting on node157.hpc.local>>
[obki@node157 ~]$ module load python/3.6.3 gcc/8.3.1
[obki@node157 ~]$ pip3 install jupyter --user

At this point Jupyter should be installed via pip3. To make sure that it installed properly, run the following command.

[obki@node157 ~]$ ls $HOME/.local/bin/jupyter
/home/obki/.local/bin/jupyter

This regular file should be the output of the ls command. Next, we can start the the Jupyter host from an interactive session using the following command.

[obki@node157 ~]$ $HOME/.local/bin/jupyter notebook --ip `hostname`

[I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki
[I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at:
[I 16:13:48.696 NotebookApp] http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.696 NotebookApp]  or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 16:13:48.707 NotebookApp] No web browser found: could not locate runnable browser.
[C 16:13:48.707 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html
    Or copy and paste one of these URLs:
        http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
     or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff

Next, we have to set up an SSH tunnel to connect to the webpage.

Setup SSH Tunnel on macOS/Linux Clients

If you have a Windows laptop/desktop, see the section below.

To setup an SSH tunnel to the Jupyter notebook session that was previously created, on a macOS or Linux client, use the following steps.

Step 1: Create the tunnel using the following command:

ssh -L 8888:node157:8888 <your_user_name>@consign.pmacs.upenn.edu

NOTE 1: Since the remote interactive session was assigned to "node157" the above SSH forwarding command must be set to same node name. Change the node name as needed.

Step 2: Click the localhost specific link generated in the output of the "jupyter notebook --ip `hostname`" command (the link that starts with "http:// 127.0.0.1").

In the above example, that would be:

http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff

NOTE 2: The output of the "jupyter notebook --ip `hostname`" command produces a unique URL with a unique token. Ensure that you are clicking the link in your specific command output. DO NOT bookmark/save this link for future use.

Setup SSH Tunnel on Windows Clients

First, open the "Tunnel" feature on MobaXterm

TunnelEdit.png


Create a new tunnel

Tunnel1.PNG


We'll be using local port forwarding.

Note: The <remote server> field is dependent on what node your interactive session (and Jupyter) is running. For me this is Node157 but may be different for you. Please adjust your input according to the node that you are running Jupyter on

These are the inputs you should use for each input field.

Tunnel2.png


Save your tunnel. Finally, start your tunnel by hitting the "Play" button, or simply by clicking "Start all tunnels"

Tunnel3.PNG


Congratulations! You've now set up a Jupyter Notebook. Go to a webpage and type localhost:8888 into the URL to access the Jupyter login page. You'll need a token to login, which you can find from the output of the last command you entered into your interactive session.

[I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki
[I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at:
[I 16:13:48.696 NotebookApp] http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.696 NotebookApp]  or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 16:13:48.707 NotebookApp] No web browser found: could not locate runnable browser.
[C 16:13:48.707 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html
    Or copy and paste one of these URLs:
        http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
     or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff

So for in my case, my token is fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff

Notes

Keep in mind that the Jupyter Notebook only lasts as long as an interactive session does, so 8 hours. To run Jupyter again in the future, simply run $HOME/.local/bin/jupyter notebook --ip `hostname` from an interactive session.

Always remember to edit your SSH tunnel to match the node from which Jupyter is running from.

Other Pages