At the HPDA frontend node of SQUID, a Jupyter Notebook environment is provided. By logging into the HPDA frontend node and launching the jupyter container, you can edit and execute programs in Python and other programming languages in a browser. This page explains how to launch the jupyter container and how to access it via a web browser.

Note: The Python used in the Jupyter Notebook environment is different from the Python environment available in SQUID. It is recommended to install packages separately for each.

Available Jupyter Containers

Container Name	Available Kernel
jupyter	python 3
jupyter-julia	python 3 julia 1.6.1
jupyter-r	python 3 R 4.1.2

Usage Guide

1.Login to HPDA Frontend

To use Jupyter Notebook, you need to connect to the HPDA frontend. Please note that the connection destination is different from the HPC frontend. The connection destination is squidhpda.hpc.cmc.osaka-u.ac.jp.

squidhpda.hpc.cmc.osaka-u.ac.jp

2.Acquiring the jupyter container image

Retrieve the jupyter container image from the local registry by executing the following command in the home directory.

$ singularity pull jupyter.sif oras://cntm:5000/master_image/[container name]:1.0

The container name part should be changed to match the container to be activated. For [container name], please click here.

example）the case of jupyter-julia container
$ singularity pull jupyter.sif oras://cntm:5000/master_image/jupyter-julia:1.0

If the container image is successfully acquired, a file name [container name].sif will be created in the home directory. If a file with the same name already exists, delete it beforehand before executing the above command. *Execution of the command will fail.

3.Starting the jupyter container

In the home directory of the logged-in user, execute the jupyter container startup shell script to start the jupyter container.

container name	execution command
jupyter	run_jupyter_container.sh
jupyter-julia	run_jupyter_container.sh -k julia
jupyter-r	run_jupyter_container.sh -k R

If the jupyter container is successfully started, the terminal will display the following:

jupyter Notebook URL：https:// squidhpda1.hpc.cmc.osaka-u.ac.jp:10125
INFO: Converting SIF file to temporary sandbox...
INFO: instance started successfully
jupyter login token : 3c4b9f3fb424b9059d1076c559adff837f9fc9ee8ce81842

The information displayed is as follows It is used in the next step "4. Accessing with a Web Browser".

jupyter Notebook URL
The URL for accessing jypyter Notebook with a web browser. The port number is automatically assigned to an available number within the range of 10000~11000.

jupyter login token
A token to be entered on the jupyter authentication screen when accessing from a web browser.

4.Access via Web Browser

Launch a web browser and access the "jupyter Notebook URL" displayed when starting the jupyter container.

You will be prompted for a user name and password, enter the following values to log in.

1)User name：SQUID User ID（User name used when logging into the HPDA front end）
2)password：SQUID password（password used when logging into the HPDA front end）

If the password authentication succeeds, you will be asked to enter the jupyter token, so enter the "jupyter login token" that was displayed in "3. Start jupyter container".

If the authentication with the jupyter token is also successful, the initial jupyter Notebook screen will appear. You are now ready to use jupyter notebook and can edit and run your program on the HPDA front end through your web browser.

5.jStopping the jupyter container

When you finish using it, execute the jupyter container stop shell script in the home directory of the logged-in user to stop the jupyter container.

Container name	execution command
jupyter	stop_jupyter_container.sh
jupyter-julia	stop_jupyter_container.sh -k julia
jupyter-r	stop_jupyter_container.sh -k R

[If you are unable to stop the jupyter container]
Both SQUID's HPC front-end servers and HPDA front-end servers are configured with four units. squidhpda1 to squidhpda4 exist for the HPDA front-end servers.
The shutdown process must be performed on the same front-end server as the startup process. In this case, log in to squidhpda2.
ssh squidhpda2
If you are unsure of the front-end server at startup, please check the ~/.jupyter.info file. If you are unable to stop the system, please contact us using the contact form.

Customizing the jupyter container

The initial jupyter container is provided in a state where components such as libraries are not installed so that users can customize it according to their preferences. This section explains how to install additional components in the jupyter container.

For example, the installation methods for numpy and matplotlib are introduced.

installing numpy

1.terminal startup

Work from the initial screen of jupyter Notebook as shown in "4. Accessing the jupyter container. Click the New button in the upper right corner of the initial screen and select a terminal.

2.Execute pip command

Type the following command into the terminal and press the Enter key.

pip3 install numpy

3.Confirmation of installation

When the installation is complete, the following message will be displayed.
The "x.x.x" part shows the version of numpy installed.

Successfully installed numpy-x.x.x

installing matplotlib

1.Start the terminal

In "4. Accessing the jupyter container," work from the initial screen of jupyter Notebook. Click the New button in the upper right corner of the initial screen and select a terminal.

2.Execute pip command

Type the following command into the terminal and press the Enter key.

pip3 install matplotlib

3.Confirmation of installation

When the installation is complete, the following message will be displayed.
The "x.x.x" part shows the version of the installed matplotlib and dependent packages.

Successfully installed cycler-x.x.x kiwisolver-x.x.x matplotlib-x.x.x pillow-x.x.x

Mounting the work area

When starting the jupyter container, a directory named "jupyter_notebook" is created in the home directory. Although only this directory can be accessed on Jupyter Notebook, you can mount other directories such as the work area under "jupyter_notebook" by following the steps below.

1.Creating a directory for work area mount

Create a directory for mounting the work area under the home jupyter_notebook directory.

mkdir ~/jupyter_notebook/user_work

2.Copy the startup script

Copy the startup script of jupyter to your home directory because it needs to be edited. There is no name specified, but in the following example, it is copied as run_jupyter_container_custom.sh.

cp /system/tool/bin/run_jupyter_container.sh ~/run_jupyter_container_custom.sh

3. Edit the startup script

Add one line to lines 75-78 of the copied startup script using an editor such as vi or emacs.

[Before modification]

singularity instance start -w -f \

--bind /etc/pki:/etc/pki/certs \

--net --network-args "portmap=$port:10443/tcp" \

${SIFNAME} ${INSTNAME}

[After modification]

singularity instance start -w -f \

--bind /etc/pki:/etc/pki/certs \

--bind [mount path]:/root/jupyter_notebook/user_work \

--net --network-args "portmap=$port:10443/tcp" \

${SIFNAME} ${INSTNAME}

The third line is added. In the [mount destination path] field, enter the path of the directory you want to access from Jupyter Notebook, such as the path of the work area. The following is an example of a user with the group name G12345 and user number a12345 mounting his own work area.

--bind /sqfs/work/G12345/a12345:/root/jupyter_notebook/user_work \

4.Start the jupyter container

Execute the edited startup script to start the jupyter container. By specifying the -k option at startup, you can select the container to start as in the normal case. The following is the command to start the jupyter container (jupyter-julia) of the Julia kernel with the edited script file name "run_jupyter_container_custom.sh".

./run_jupyter_container_custom.sh -k julia

The following steps are the same as for normal use. Please see here.