Jupyter-JSC

Interactive supercomputing based on UNICORE

Research and analysis of large amounts of data from scientific simulations, in-situ visualization and application control are convincing scenarios for interactive supercomputing. The open-source software Jupyter¹ is a tool that has already been successfully used in many scientific disciplines. With its open and flexible web-based design, Jupyter (or JupyterLab²) is ideal for combining a wide variety of workflows and programming methods under one interface. With the multi-user capability of Jupyter via JuypterHub³, it is suitable for use at supercomputing centers and thus combines the way HPC users work on the local laptop with the operation of a supercomputer. To ensure that each user has access to his data on the HPC systems UNICORE is used. The combination of the UNICORE tools XUUDB, UNICORE/X, Gateway and TSI allows JupyterHub to offer users to spawn Jupyter applications with their HPC accounts on all supported HPC systems. The integration of Unity-IdM⁴ into UNICORE enables Jupyter-JSC⁵ to delegate the login procedure to Unity-Idm. Therefore Jupyter-JSC is able to meet the desire for more interactivity in supercomputing and to open up new possibilities of high-performance computing. A simple, direct web access for starting and connecting with Jupyter or JupyterLab to the login or compute node was integrated at the Juelich Supercomputing Centre (JSC).

UNICORE enables Jupyter-JSC to start and manage jobs on the HPC systems at JSC. This ensures that every user has access to his data and can work with a Jupyter application on them. In addition, UNICORE offers reliable management of the Jupyter application. It provides access to all information output by the Jupyter application. Thanks to UNICORE, JupyterHub does not require direct access to the user’s HPC account. In practice Jupyter-JSC could not be operated without UNICORE.

References

[1] Jupyter: jupyter.org
[2] JupyterLab: github.com/jupyterlab/jupyterlab
[3] JupyterHub: github.com/jupyterhub/jupyterhub
[4] Unity project website: unity-idm.eu
[5] Jupyter-JSC: jupyter-jsc.fz-juelich.de