Difference between revisions of "HPC:Jupyter"
| (19 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
===Jupyter Notebook=== | ===Jupyter Notebook=== | ||
| − | Currently, the HPC is capable of hosting a Jupyter Notebook workspace in an interactive session. | + | Currently, the HPC is capable of hosting a Jupyter Notebook workspace in an interactive session. This limits the Jupyter Notebook to an 8 hour session. Please follow these steps to set up your Jupyter Notebook workspace. |
The initial setup for Jupyter is a little tricky, but once completed is easy to get up and running again. | The initial setup for Jupyter is a little tricky, but once completed is easy to get up and running again. | ||
| − | This guide will assume that you are using MobaXTerm as your terminal emulator in order to set up SSH tunneling. However, this can also be done via other tools (Powershell, PuTTy, et cetera) but those tools will not be covered in this guide. If you have questions about setting up Jupyter with other tools, please contact the HPC team (or come to office hours!) | + | This guide will assume that you are using Windows 10 MobaXTerm as your terminal emulator in order to set up SSH tunneling. However, this can also be done via other tools (Powershell, PuTTy, et cetera) but those tools will not be covered in this guide. If you have questions about setting up Jupyter with other tools, please contact the HPC team (or come to office hours!) |
First, install the Jupyter client. | First, install the Jupyter client. | ||
| Line 10: | Line 10: | ||
[obki@consign ~]$ bsub -Is bash | [obki@consign ~]$ bsub -Is bash | ||
<< Waiting for dispatch...>> | << Waiting for dispatch...>> | ||
| − | <<Starting on | + | <<Starting on node157.hpc.local>> |
| − | [obki@ | + | [obki@node157 ~]$ module load python/3.6.3 gcc/8.3.1 |
| − | [obki@ | + | [obki@node157 ~]$ pip3 install jupyter --user |
</pre> | </pre> | ||
At this point Jupyter should be installed via pip3. To make sure that it installed properly, run the following command. | At this point Jupyter should be installed via pip3. To make sure that it installed properly, run the following command. | ||
<pre> | <pre> | ||
| − | [obki@ | + | [obki@node157 ~]$ ls $HOME/.local/bin/jupyter |
/home/obki/.local/bin/jupyter | /home/obki/.local/bin/jupyter | ||
</pre> | </pre> | ||
| − | This regular file should be the output of the <code>ls</code> command. Next, we can start the the Jupyter host from an interactive session | + | This regular file should be the output of the <code>ls</code> command. Next, we can start the the Jupyter host from an interactive session using the following command. |
<pre> | <pre> | ||
| − | [obki@ | + | [obki@node157 ~]$ $HOME/.local/bin/jupyter notebook --ip `hostname` |
| + | |||
[I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki | [I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki | ||
[I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at: | [I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at: | ||
| − | [I 16:13:48.696 NotebookApp] http:// | + | [I 16:13:48.696 NotebookApp] http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff |
[I 16:13:48.696 NotebookApp] or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff | [I 16:13:48.696 NotebookApp] or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff | ||
[I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). | [I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). | ||
| Line 33: | Line 34: | ||
file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html | file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html | ||
Or copy and paste one of these URLs: | Or copy and paste one of these URLs: | ||
| − | http:// | + | http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff |
or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff | or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff | ||
</pre> | </pre> | ||
| − | + | Next, we have to set up an SSH tunnel to connect to the webpage. | |
| − | + | ||
| + | ====Setup SSH Tunnel on macOS/Linux Clients==== | ||
| + | If you have a Windows laptop/desktop, see the section [[HPC:Jupyter#Setup_SSH_Tunnel_on_Windows_Clients|below]]. | ||
| + | |||
| + | To setup an SSH tunnel to the Jupyter notebook session that was previously created, '''on a macOS or Linux''' client, use the following steps. | ||
| + | |||
| + | Step 1: Create the tunnel using the following command: | ||
| + | |||
| + | <pre> | ||
| + | ssh -L 8888:node157:8888 <your_user_name>@consign.pmacs.upenn.edu | ||
| + | </pre> | ||
| + | |||
| + | '''NOTE 1:''' Since the remote interactive session was assigned to "node157" the above SSH forwarding command must be set to same node name. '''Change the node name as needed'''. | ||
| − | + | Step 2: '''Click the localhost specific link''' generated in the output of the "jupyter notebook --ip `hostname`" command (the link that starts with "http:// 127.0.0.1"). | |
| + | |||
| + | In the above example, that would be: | ||
| + | |||
| + | <pre> | ||
| + | http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff | ||
| + | </pre> | ||
| + | |||
| + | '''NOTE 2:''' The output of the "jupyter notebook --ip `hostname`" command produces a '''unique''' URL with a '''unique''' token. Ensure that you are clicking the link in your specific command output. '''DO NOT''' bookmark/save this link for future use. | ||
| + | |||
| + | ====Setup SSH Tunnel on Windows Clients==== | ||
First, open the "Tunnel" feature on MobaXterm | First, open the "Tunnel" feature on MobaXterm | ||
| − | [[image: | + | <br /> |
| + | [[image:TunnelEdit.png|center|600px]] | ||
| + | <br /> | ||
Create a new tunnel | Create a new tunnel | ||
| + | <br /> | ||
[[image:Tunnel1.PNG|center|600px]] | [[image:Tunnel1.PNG|center|600px]] | ||
| + | <br /> | ||
We'll be using local port forwarding. | We'll be using local port forwarding. | ||
| − | <strong>Note: The <code><remote server></code> field is | + | <strong>Note: The <code><remote server></code> field is dependent on what node your interactive session (and Jupyter) is running. For me this is <code>Node157</code> but may be different for you. Please adjust your input according to the node that you are running Jupyter on</strong> |
These are the inputs you should use for each input field. | These are the inputs you should use for each input field. | ||
| + | <br /> | ||
[[image:Tunnel2.png|center|600px]] | [[image:Tunnel2.png|center|600px]] | ||
| + | <br /> | ||
Save your tunnel. Finally, start your tunnel by hitting the "Play" button, or simply by clicking "Start all tunnels" | Save your tunnel. Finally, start your tunnel by hitting the "Play" button, or simply by clicking "Start all tunnels" | ||
| − | [[image: | + | <br /> |
| + | [[image:Tunnel3.PNG|center|600px]] | ||
| + | <br /> | ||
| + | Congratulations! You've now set up a Jupyter Notebook. Go to a webpage and type <code>localhost:8888</code> into the URL to access the Jupyter login page. You'll need a token to login, which you can find from the output of the last command you entered into your interactive session. | ||
| + | <pre> | ||
| + | [I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki | ||
| + | [I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at: | ||
| + | [I 16:13:48.696 NotebookApp] http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff | ||
| + | [I 16:13:48.696 NotebookApp] or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff | ||
| + | [I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). | ||
| + | [W 16:13:48.707 NotebookApp] No web browser found: could not locate runnable browser. | ||
| + | [C 16:13:48.707 NotebookApp] | ||
| + | |||
| + | To access the notebook, open this file in a browser: | ||
| + | file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html | ||
| + | Or copy and paste one of these URLs: | ||
| + | http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff | ||
| + | or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff | ||
| + | </pre> | ||
| + | |||
| + | So for in my case, my token is <code>fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff</code> | ||
| + | ===='''Notes'''==== | ||
| + | Keep in mind that the Jupyter Notebook only lasts as long as an interactive session does, so 8 hours. To run Jupyter again in the future, simply run <code>$HOME/.local/bin/jupyter notebook --ip `hostname`</code> from an interactive session. | ||
| + | |||
| + | Always remember to edit your SSH tunnel to match the node from which Jupyter is running from. | ||
| + | |||
| + | === Other Pages === | ||
| + | ---- | ||
| + | <div class="mw-collapsible mw-collapsed"> | ||
| + | *[[HPC:Main_Page|HPC Main Page]] | ||
| + | *[[HPC:FAQ|HPC FAQ ]] | ||
| + | *[[HPC:Login|Connecting to the PMACS cluster]] | ||
| + | *[[HPC:Software|Available Software]] | ||
| + | *[[HPC:Archive System|PMACS Archive System]] | ||
| + | </div> | ||
Latest revision as of 19:37, 21 November 2023
Contents
Jupyter Notebook
Currently, the HPC is capable of hosting a Jupyter Notebook workspace in an interactive session. This limits the Jupyter Notebook to an 8 hour session. Please follow these steps to set up your Jupyter Notebook workspace.
The initial setup for Jupyter is a little tricky, but once completed is easy to get up and running again.
This guide will assume that you are using Windows 10 MobaXTerm as your terminal emulator in order to set up SSH tunneling. However, this can also be done via other tools (Powershell, PuTTy, et cetera) but those tools will not be covered in this guide. If you have questions about setting up Jupyter with other tools, please contact the HPC team (or come to office hours!)
First, install the Jupyter client.
[obki@consign ~]$ bsub -Is bash << Waiting for dispatch...>> <<Starting on node157.hpc.local>> [obki@node157 ~]$ module load python/3.6.3 gcc/8.3.1 [obki@node157 ~]$ pip3 install jupyter --user
At this point Jupyter should be installed via pip3. To make sure that it installed properly, run the following command.
[obki@node157 ~]$ ls $HOME/.local/bin/jupyter /home/obki/.local/bin/jupyter
This regular file should be the output of the ls command. Next, we can start the the Jupyter host from an interactive session using the following command.
[obki@node157 ~]$ $HOME/.local/bin/jupyter notebook --ip `hostname`
[I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki
[I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at:
[I 16:13:48.696 NotebookApp] http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.696 NotebookApp] or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 16:13:48.707 NotebookApp] No web browser found: could not locate runnable browser.
[C 16:13:48.707 NotebookApp]
To access the notebook, open this file in a browser:
file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html
Or copy and paste one of these URLs:
http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
Next, we have to set up an SSH tunnel to connect to the webpage.
Setup SSH Tunnel on macOS/Linux Clients
If you have a Windows laptop/desktop, see the section below.
To setup an SSH tunnel to the Jupyter notebook session that was previously created, on a macOS or Linux client, use the following steps.
Step 1: Create the tunnel using the following command:
ssh -L 8888:node157:8888 <your_user_name>@consign.pmacs.upenn.edu
NOTE 1: Since the remote interactive session was assigned to "node157" the above SSH forwarding command must be set to same node name. Change the node name as needed.
Step 2: Click the localhost specific link generated in the output of the "jupyter notebook --ip `hostname`" command (the link that starts with "http:// 127.0.0.1").
In the above example, that would be:
http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
NOTE 2: The output of the "jupyter notebook --ip `hostname`" command produces a unique URL with a unique token. Ensure that you are clicking the link in your specific command output. DO NOT bookmark/save this link for future use.
Setup SSH Tunnel on Windows Clients
First, open the "Tunnel" feature on MobaXterm
Create a new tunnel
We'll be using local port forwarding.
Note: The <remote server> field is dependent on what node your interactive session (and Jupyter) is running. For me this is Node157 but may be different for you. Please adjust your input according to the node that you are running Jupyter on
These are the inputs you should use for each input field.
Save your tunnel. Finally, start your tunnel by hitting the "Play" button, or simply by clicking "Start all tunnels"
Congratulations! You've now set up a Jupyter Notebook. Go to a webpage and type localhost:8888 into the URL to access the Jupyter login page. You'll need a token to login, which you can find from the output of the last command you entered into your interactive session.
[I 16:13:48.696 NotebookApp] Serving notebooks from local directory: /home/obki
[I 16:13:48.696 NotebookApp] The Jupyter Notebook is running at:
[I 16:13:48.696 NotebookApp] http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.696 NotebookApp] or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
[I 16:13:48.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 16:13:48.707 NotebookApp] No web browser found: could not locate runnable browser.
[C 16:13:48.707 NotebookApp]
To access the notebook, open this file in a browser:
file:///home/obki/.local/share/jupyter/runtime/nbserver-19940-open.html
Or copy and paste one of these URLs:
http://node157.hpc.local:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
or http://127.0.0.1:8888/?token=fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
So for in my case, my token is fbf40f23ca994ac5046337e6427ffea8d2be346fa54653ff
Notes
Keep in mind that the Jupyter Notebook only lasts as long as an interactive session does, so 8 hours. To run Jupyter again in the future, simply run $HOME/.local/bin/jupyter notebook --ip `hostname` from an interactive session.
Always remember to edit your SSH tunnel to match the node from which Jupyter is running from.
