Using DXJupyterLab

Getting Started Developers Downloads Science Corner
Search…
Using DXJupyterLab
Use Jupyter notebooks on the DNAnexus Platform to craft sophisticated custom analyses in your preferred coding language.
A license is required to access DXJupyterLab. Contact [email protected] for more information.
​Jupyter notebooks are a popular way to track the work performed in computational experiments the way a lab notebook tracks the work done in a wet lab setting. DXJupyterLab is an application provided by DNAnexus that allows you to perform computational experiments in the DNAnexus cloud using Jupyter notebooks. DXJupyterLab allows users on the DNAnexus platform to collaborate on notebooks and extends JupyterLab with options for directly accessing a DNAnexus project from the JupyterLab environment.
Why Use DXJupyterLab?
DXJupyterLab supports the use of Bioconductor and Bioconda, useful tools for bioinformatics analysis.
DXJupyterlab is a versatile application that can be used to:
Collaborate on exploratory analysis of data
Reproduce and fork work performed in computational analyses
Visualize and gain insights into data generated from biological experiments
Create figures and tables for scientific publications
Build and test algorithms directly in the cloud before creating DNAnexus apps and workflows
Test and train machine/deep learning models
Interactively run commands on a terminal
JupyterLab can be run on the DNAnexus platform in two different ways: a general-purpose JupyterLab application and a Spark cluster-enabled one, which can be used within the DNAnexus Apollo framework.
Both apps instantiate a JupyterLab server that allows for data analyses to be interactively performed in Jupyter notebooks on a DNAnexus worker.
The DXJupyterLab Spark Cluster app contains all the features found in the general-purpose DXJupyterLab along with access to a fully-managed, on-demand Spark cluster for big data processing and translational informatics.
Creating Interactive Notebooks
A step-by-step guide on how to start with DXJupyterLab and create and edit Jupyter notebooks can be found in the Quickstart.
DXJupyterLab Environments
Creating a DXJupyterlab session requires the use of two different environments:
1.
The DNAnexus project (accessible through the web platform and the CLI)
2.
The worker execution environment
The project on the DNAnexus platform
You have direct access to the project in which the application is run from the JupyterLab session. The project file browser (which lists folders, notebooks, and other files in the project) can be accessed from the DNAnexus tab in the left sidebar or from the terminal:
The project is selected when the DXJupyterLab app is started and cannot be subsequently changed.
You can create, edit, and save notebooks directly in the DNAnexus project as well as duplicate, delete, or download them to your local machine. These notebooks are displayed under the DNAnexus tab on the left of the DxJupyterLab app and are fetched from and saved to the project on the DNAnexus platform without being stored in the DxJupyterLab execution environment file system. These notebooks persist in the DNAnexus project after the DXJupyterLab instance is terminated.
The project file browser displays all subfolders, all Jupyter notebooks, and files in the project (limited to the 1,000 most recently modified files). In the Spark-enabled app databases in the project are also visible (limited to the 1,000 most recently modified databases). A list of all the objects in the project can be obtained programmatically or by using dx ls. The file listing is refreshed every 10s and it is possible to enforce a refresh by clicking on the whirl arrow icon in the top right corner of the file browser.
Worker execution environment
When you open and run a notebook from the DNAnexus file browser the kernel corresponding to this notebook is started in the worker execution environment and will be used to execute the notebook code. DNAnexus notebook can be recognized by the [DX] that is prepended to its name in the tab of all opened notebooks.
The execution environment file browser can be accessed from the left sidebar (notice the folder icon at the top) or from the terminal:
You can also create Jupyter notebooks in the worker execution environment through the File menu. These notebooks are stored on the local file system of the DxJupyterLab execution environment and have to be persisted in a DNAnexus project. More information about saving can be found in the next section.
Accessing input data and saving output data
DxJupyterLab is an application that runs in a separate worker execution environment. Any outputs from the notebook will be generated in this temporary execution environment and should be persisted in a DNAnexus project or be lost when the DxJupyterLab session ends.
Files can be persisted in the DNAnexus project by using one of these options:
dx upload in bash console
drag'n'drop the file onto the DNAnexus listing (the file will be uploaded to the current DNAnexus folder)
use DNAnexus button in the menu, choose Upload Local File and select the file which you want to upload (file will be uploaded to the current DNAnexus folder)
The input data from DNAnexus project can be accessed from a Jupyter notebook in one of the following ways.
For reading the input file multiple times or for reading a large fraction of the file in random order:
Download the file from the DNAnexus project to the execution environment with dx downloadand access the downloaded local file from Jupyter notebook.
For scanning the content of the input file once or for reading only a small fraction of file's content:
A project in which the app is running is mounted in a read-only fashion at/mnt/project folder.  Reading the content of the files in /mnt/project dynamically fetches the content from the DNAnexus platform, so this method uses minimal disk space in the JupyterLab execution environment, but uses more api calls to fetch the content.
Exporting DNAnexus notebook
Exporting DNAnexus notebooks (e.g. to HTML or PDF) is not supported, but it's possible to dx download the DNAnexus notebook from the current DNAnexus project to the JupyterLab environment and export the downloaded notebook. For exporting local notebook to certain formats, the following commands might be needed beforehand: apt-get update && apt-get install texlive-xetex texlive-fonts-recommended texlive-plain-generic.
Non-interactive execution of notebooks
A command can be executed in the DxJupyterLab worker execution environment without starting an interactive JupyterLab server. To do that, provide the cmd input and additional input files using the in input file array to the DxJupyterLab app. The provided command will run in the /opt/notebooks/directory and any output files generated in this directory will be uploaded to the project and returned in the out output field of the job that ran DxJupyterLab app.
The cmd input makes it possible to use the papermillcommand that is pre-installed in the DxJupyterLab environment to execute notebooks non-interactively. For example, to execute all the cells in a notebook and produce an output notebook:
1
my_cmd="papermill notebook.ipynb output_notebook.ipynb"
2
dx run dxjupyterlab -icmd="$my_cmd" -iin="notebook.ipynb"
Copied!
Where notebook.ipynb is the input notebook to thepapermillcommand, which is passed to the dxjupyterlab app using thein input, and output_notebook.ipynb is the name of the output notebook, which will contain the result of executing the input notebook and will be uploaded to the project at the end of app's execution. See the DxJupyterLab app page for details.
Collaboration in the Cloud
Collaborators can work on notebooks in the project without the risk of overwriting each other's changes.
Notebook locking during editing
If a user has opened a specific notebook in a JupyterLab session, other users cannot open or edit the notebook. This is indicated by a red lock icon next to the notebook's name.
It is still possible to create a duplicate to see what changes are being saved in the locked notebook or to continue work on this "forked" version of the notebook. To copy a notebook, right-click on its name and select Duplicate; after a few seconds, a notebook with the same name and a "copy" suffix should appear in the project.
Once the editing user closes the notebook, the lock will be released and anybody else with access to the project will be able to open it.
Notebook versioning
Whenever a notebook is saved in the project, it is uploaded to the platform as a new file that will replace the previous version, i.e. the file of the same name. The previous version is moved to the .Notebook_archive folder with a timestamp suffix added to its name and its ID is saved in the properties of the new file. Saving notebooks directly in the project ensures that your analyses won't be lost when the DXJupyterLab session ends.
Session Timeout Control
DXJupyterLab sessions begin with a set duration, after which they will shut down automatically. The timeout clock is displayed in the footer on the right side and it can also be adjusted there (using the Update duration button). Even if the DxJupyterLab webpage is closed, the termination will be executed at the set timestamp. Job lengths have an upper limit of 30 days, which cannot be extended.
A session can be terminated immediately from the top menu (DNAnexus > End Session).
Environment Snapshots
It is possible to save the current session environment and data and reload it later by creating a session snapshot (DNAnexus > Create Snapshot).
A DXJupyterLab session is run in a Docker container, and a session snapshot file is a tarball generated by saving the Docker container state (with the docker commit and docker save commands). Any installed packages and files created locally are saved to a snapshot file, with the exception of directories /home/dnanexus and /mnt/, which are not included. This file is then uploaded to the project to .Notebook_snapshots and can be passed as input the next time the app is started.
NOTE: If many large files are created locally, the resulting snapshots will take very long to save and load. In general, it is recommended not to snapshot more than 1 GB of locally saved data/packages and rely on downloading larger files as needed.
Viewing Other Files in the Project
Viewing any other file types from your project, such as CSV, JSON, PDF files, images, scripts, etc. is convenient because JupyterLab will render them accordingly. For example, JSON files will be collapsible and easy to navigate and CSV files will be presented in the tabular format.
However, editing and saving any open files from the project other than IPython notebooks will result in an error.
Permissions in the JupyterLab Session
The JupyterLab apps are run in a specific project, defined at start time, and this project cannot be subsequently changed. The job associated with the JupyterLab app has CONTRIBUTE access to the project in which it is run.
When running DXJupyterLab app, it is possible to view, but not update, other projects the user has access to. This enhanced scope is required to be able to read databases which may be located in different projects and are not cloneable.
Running Jobs in the JupyterLab Session
It is possible to start new jobs with dx run from a notebook or the Terminal. If thebillToof the project that the JupyterLab session is running in is not licensed to start detached executions, the started jobs will be the subjobs of the interactive JupyterLab session. In this case, the --project argument todx runwill be ignored and the JupyterLab job's workspace is used to run the job, not the given project. Also, if the attached subjob fails or is terminated on the DNAnexus platform, the whole job tree will be terminated, including the interactive JupyterLab session.
Environment and Pre-Installed Packages
The DXJupyterLab app is a Docker-based app that runs the JupyterLab server instance in a Docker container. The server runs on port 443. Because the app is an HTTPS app, you can bring up the JupyterLab environment in a web browser using the URL https://job-xxxx.dnanexus.cloud, where job-xxxx is the ID of the job that runs the app. Only the user who launched the JupyterLab job has access to the JupyterLab environment. Other users will see a “403 Permission Forbidden” message under the JupyterLab session's URL.
NOTE: The JupyterLab server runs in the Miniconda Python 3.6 environment and Ubuntu:18.04 is the container's operating system.
In addition to dx-toolkit, the JupyterLab server comes pre-installed with a number of popular data packages:
numpy
pandas
scipy
seaborn
matplotlib
Kernels available for the notebooks:
Python 3.6
R 4.1.1
CUDA libraries and drivers are installed on GPU-enabled instances.
Additional packages can easily be installed during a JupyterLab session. By creating a Docker container snapshot, users can then start subsequent sessions with the new packages pre-installed by providing the snapshot as input.
JupyterLab Documentation
You can find more information on the features and benefits of JupyterLab in the official JupyterLab documentation.
Next Steps
Create your first notebooks by following Quickstart.
​References has tips and info on the most useful DXJupyterLab features.