Using DXJupyterLab
Use Jupyter notebooks on the DNAnexus Platform to craft sophisticated custom analyses in your preferred coding language.
Jupyter notebooks are a popular way to track the work performed in computational experiments the way a lab notebook tracks the work done in a wet lab setting. DXJupyterLab, or JupyterLab, is an application provided by DNAnexus that allows you to perform computational experiments in the DNAnexus cloud using Jupyter notebooks. DXJupyterLab allows users on the DNAnexus platform to collaborate on notebooks and extends JupyterLab with options for directly accessing a DNAnexus project from the JupyterLab environment.
DXJupyterLab supports the use of Bioconductor and Bioconda, useful tools for bioinformatics analysis.
DXJupyterlab is a versatile application that can be used to:
- Collaborate on exploratory analysis of data
- Reproduce and fork work performed in computational analyses
- Visualize and gain insights into data generated from biological experiments
- Create figures and tables for scientific publications
- Build and test algorithms directly in the cloud before creating DNAnexus apps and workflows
- Test and train machine/deep learning models
- Interactively run commands on a terminal
JupyterLab can be run on the DNAnexus platform in two different ways: a general-purpose JupyterLab application and a Spark cluster-enabled one, which can be used within the DNAnexus Apollo framework.
Both apps instantiate a JupyterLab server that allows for data analyses to be interactively performed in Jupyter notebooks on a DNAnexus worker.
The DXJupyterLab Spark Cluster app contains all the features found in the general-purpose DXJupyterLab along with access to a fully-managed, on-demand Spark cluster for big data processing and translational informatics.
A step-by-step guide on how to start with DXJupyterLab and create and edit Jupyter notebooks can be found in the Quickstart.
Creating a DXJupyterlab session requires the use of two different environments:
- 1.The DNAnexus project (accessible through the web platform and the CLI).
- 2.The worker execution environment.
You have direct access to the project in which the application is run from the JupyterLab session. The project file browser (which lists folders, notebooks, and other files in the project) can be accessed from the DNAnexus tab in the left sidebar or from the terminal:
The project is selected when the DXJupyterLab app is started and cannot be subsequently changed.
The project file browser displays all subfolders, all Jupyter notebooks, and files in the project (limited to the 1,000 most recently modified files). In the Spark-enabled app databases in the project are also visible (limited to the 1,000 most recently modified databases). A list of all the objects in the project can be obtained programmatically or by using
dx ls. The file listing is refreshed every 10s and it is possible to enforce a refresh by clicking on the whirl arrow icon in the top right corner of the file browser.When you open and run a notebook from the DNAnexus file browser the kernel corresponding to this notebook is started in the worker execution environment and will be used to execute the notebook code. DNAnexus notebooks will have a
[DX] prepended to the notebook name in the tab of all opened notebooks.
The execution environment file browser can be accessed from the left sidebar (notice the folder icon at the top) or from the terminal:
You can also create Jupyter notebooks in the worker execution environment through the File menu. These notebooks are stored on the local file system of the DxJupyterLab execution environment and have to be persisted in a DNAnexus project. More information about saving can be found in the next section.
You can create, edit, and save notebooks directly in the DNAnexus project as well as duplicate, delete, or download them to your local machine. Notebooks stored in your DNAnexus project, which are housed within the DNAnexus tab on the left sidebar, are fetched from and saved to the project on the DNAnexus platform without being stored in the JupyterLab execution environment file system. These are referred to as “DNAnexus notebooks” and these notebooks persist in the DNAnexus project after the DXJupyterLab instance is terminated.
DNAnexus notebooks can be recognized by the
[DX] that is prepended to its name in the tab of all opened notebooks.DNAnexus notebooks can be created by clicking the DNAnexus Notebook icon from the Launcher tab that appears upon starting the JupyterLab session, or by clicking the DNAnexus tab on the upper menu and then clicking “New notebook”. The Launcher tab can also be opened by clicking File and then selecting “New Launcher” from the upper menu.
To create a new local notebook, click the File tab in the upper menu and then select “New" and then “Notebook”. These non-DNAnexus notebooks can be saved to DNAnexus by simply dragging and dropping them in the DNAnexus file viewer in the left panel.
In JupyterLab, users can access input data that is located in a DNAnexus project in one of the following ways.
- For reading the input file multiple times or for reading a large fraction of the file in random order:
- Download the file from the DNAnexus project to the execution environment with
dx downloadand access the downloaded local file from Jupyter notebook.
- For scanning the content of the input file once or for reading only a small fraction of file's content:
- A project in which the app is running is mounted in a read-only fashion at
/mnt/projectfolder. Reading the content of the files in/mnt/projectdynamically fetches the content from the DNAnexus platform, so this method uses minimal disk space in the JupyterLab execution environment, but uses more api calls to fetch the content.
Files such as local notebooks can be persisted in the DNAnexus project by using one of these options:
dx uploadin bash console.- Drag and drop the file onto the DNAnexus tab that is in the column of icons on the left side of the screen. This will upload the file into the currently selected DNAnexus folder.
Exporting DNAnexus notebooks (e.g. to HTML or PDF) is not supported, but it's possible to
dx download the DNAnexus notebook from the current DNAnexus project to the JupyterLab environment and export the downloaded notebook. For exporting local notebook to certain formats, the following commands might be needed beforehand: apt-get update && apt-get install texlive-xetex texlive-fonts-recommended texlive-plain-generic.A command can be executed in the DxJupyterLab worker execution environment without starting an interactive JupyterLab server. To do that, provide the
cmd input and additional input files using the in input file array to the DxJupyterLab app. The provided command will run in the /opt/notebooks/directory and any output files generated in this directory will be uploaded to the project and returned in the out output field of the job that ran DxJupyterLab app.The cmd input makes it possible to use the
papermillcommand that is pre-installed in the DxJupyterLab environment to execute notebooks non-interactively. For example, to execute all the cells in a notebook and produce an output notebook:my_cmd="papermill notebook.ipynb output_notebook.ipynb"
dx run dxjupyterlab -icmd="$my_cmd" -iin="notebook.ipynb"
Where
notebook.ipynb is the input notebook to thepapermillcommand, which is passed to the dxjupyterlab app using thein input, and output_notebook.ipynb is the name of the output notebook, which will contain the result of executing the input notebook and will be uploaded to the project at the end of app's execution. See the DxJupyterLab app page for details.Collaborators can work on notebooks in the project without the risk of overwriting each other's changes.
If a user has opened a specific notebook in a JupyterLab session, other users cannot open or edit the notebook. This is indicated by a red lock icon next to the notebook's name.
It is still possible to create a duplicate to see what changes are being saved in the locked notebook or to continue work on this "forked" version of the notebook. To copy a notebook, right-click on its name and select
Duplicate; after a few seconds, a notebook with the same name and a "copy" suffix should appear in the project.Once the editing user closes the notebook, the lock will be released and anybody else with access to the project will be able to open it.
Whenever a notebook is saved in the project, it is uploaded to the platform as a new file that will replace the previous version, i.e. the file of the same name. The previous version is moved to the
.Notebook_archive folder with a timestamp suffix added to its name and its ID is saved in the properties of the new file. Saving notebooks directly in the project ensures that your analyses won't be lost when the DXJupyterLab session ends.DXJupyterLab sessions begin with a set duration, after which they will shut down automatically. The timeout clock is displayed in the footer on the right side and it can also be adjusted there (using the
Update duration button). Even if the DxJupyterLab webpage is closed, the termination will be executed at the set timestamp. Job lengths have an upper limit of 30 days, which cannot be extended.A session can be terminated immediately from the top menu (
DNAnexus > End Session).It is possible to save the current session environment and data and reload it later by creating a session snapshot (
DNAnexus > Create Snapshot).A DXJupyterLab session is run in a Docker container, and a session snapshot file is a tarball generated by saving the Docker container state (with the
docker commit and docker save commands). Any installed packages and files created locally are saved to a snapshot file, with the exception of directories /home/dnanexus and /mnt/, which are not included. This file is then uploaded to the project to .Notebook_snapshots and can be passed as input the next time the app is started.NOTE: If many large files are created locally, the resulting snapshots will take very long to save and load. In general, it is recommended not to snapshot more than 1 GB of locally saved data/packages and rely on downloading larger files as needed.
Viewing any other file types from your project, such as CSV, JSON, PDF files, images, scripts, etc. is convenient because JupyterLab will render them accordingly. For example, JSON files will be collapsible and easy to navigate and CSV files will be presented in the tabular format.
However, editing and saving any open files from the project other than IPython notebooks will result in an error.
The JupyterLab apps are run in a specific project, defined at start time, and this project cannot be subsequently changed. The job associated with the JupyterLab app has CONTRIBUTE access to the project in which it is run.
When running DXJupyterLab app, it is possible to view, but not update, other projects the user has access to. This enhanced scope is required to be able to read databases which may be located in different projects and are not cloneable.
It is possible to start new jobs with
dx run from a notebook or the Terminal. If thebillToof the project that the JupyterLab session is running in is not licensed to start detached executions, the started jobs will be the subjobs of the interactive JupyterLab session. In this case, the --project argument todx runwill be ignored and the JupyterLab job's workspace is used to run the job, not the given project. Also, if the attached subjob fails or is terminated on the DNAnexus platform, the whole job tree will be terminated, including the interactive JupyterLab session.Jobs are limited to a runtime of 30 days. Jobs running longer than 30 days will be automatically terminated.
The DXJupyterLab app is a Docker-based app that runs the JupyterLab server instance in a Docker container. The server runs on port 443. Because the app is an HTTPS app, you can bring up the JupyterLab environment in a web browser using the URL
https://job-xxxx.dnanexus.cloud, where job-xxxx is the ID of the job that runs the app. Only the user who launched the JupyterLab job has access to the JupyterLab environment. Other users will see a “403 Permission Forbidden” message under the JupyterLab session's URL.The JupyterLab server runs in the Miniconda Python 3.6 environment and Ubuntu:18.04 is the container's operating system.
When launching Jupyterlab, the
feature options available are PYTHON_R, ML, IP, and STATA. Selecting thePYTHON_R feature (default option) loads the environment with Python3 and R kernel and interpreter. Selecting the ML feature loads the environment with Python3 and Machine Learning packages such as TensorFlow, PyTorch, CNTK as well as the Image Processing package NiPype, but it does not contain R. Selecting the IMAGE_PROCESSING feature loads the environment with Python3 and Image Processing packages such as NiPype, FreeSurfer and FSL but it does not contain R. The FreeSurfer package requires a license to run. Details about license creation and usage can be found here. The STATA feature requires a license to run.When starting the app with the PYTHON_R feature the following libraries are included in addition to
dx-toolkit,the standard Python 3.6.5 libraries, and R 4.1.3 libraries:Python libraries:
- dxpy (0.314.0)
- matplotlib (3.0.3)
- numpy (1.18.5)
- pandas (0.25.3)
- papermill (2.3.3)
- scipy (1.5.4)
- seaborn (0.10.1)
- dxdata (0.35.0) and dependencies:
- findspark (2.0.1)
- py4j (0.10.7)
- pyhive (0.6.4)
- pyspark (2.4.4)
- sasl (0.2.1)
- sqlalchemy (1.3.23)
- thrift (0.13.0)
- thrift-sasl (0.4.2)
R libraries:
- BiocManager (1.30.16)
- coloc (5.1.0)
- epiR (2.0.33)
- hyprcoloc (1.0)
- incidence (1.7.2)
- MendelianRandomization (0.5.1)
- outbreaks (1.9.0)
- prevalence (0.4.0)
When starting the app with the
STATA feature, you need a valid Stata license to run the session. See here for more information about running Stata in JupyterLab. Starting the app with STATA allows the above libraries from PYTHON_R to be included, and additionally includes:- Stata (16)
- stata_kernel (1.12.2)
Using the app with the
ML feature allows the following libraries to be included:Python libraries:
- dxpy (0.314.0)
- matplotlib (3.0.3)
- numpy (1.18.5)
- pandas (0.25.3)
- papermill (2.3.3)
- scipy (1.5.4)
- seaborn (0.10.1)
- dxdata (0.35.0) and dependencies:
- findspark (2.0.1)
- py4j (0.10.7)
- pyhive (0.6.4)
- pyspark (2.4.4)
- sasl (0.2.1)
- sqlalchemy (1.3.23)
- thrift (0.13.0)
- thrift-sasl (0.4.2)
Machine learning Python libraries:
- cntk (2.7)
- keras (2.4.3)
- scikit-learn (0.23.1)
- tensorflow (2.3.0)
- torch (1.5.1)
Using the app with the
IMAGE_PROCESSING feature allows the following libraries to be included:Python libraries:
- dxpy (0.314)
- matplotlib (3.0.3)
- numpy (1.18.5)
- pandas (0.25.3)
- papermill (2.3.3)
- scipy (1.5.4)
- seaborn (0.10.1)
- dxdata (0.35.0) and dependencies:
- findspark (2.0.1)
- py4j (0.10.7)
- pyhive (0.6.4)
- pyspark (2.4.4)
- sasl (0.2.1)
- sqlalchemy (1.3.23)
- thrift (0.13.0)
- thrift-sasl (0.4.2)
Image Processing Libraries:
- nipype (1.5.1)
- FreeSurfer (7.2.0)
- FSL (3.2.3)
CUDA libraries (10.2) and nvidia drivers (440.95.01) are installed on GPU-enabled instances.
You can find more information on the features and benefits of JupyterLab in the official JupyterLab documentation.
Last modified 16d ago