# JupyterLab Reference

{% hint style="info" %}
JupyterLab is accessible to all users of the UK Biobank Research Analysis Platform and the Our Future Health Trusted Research Environment.

A license is required to access JupyterLab on the DNAnexus Platform. [Contact DNAnexus Sales](mailto:sales@dnanexus.com) for more information.
{% endhint %}

## Download Files from the Project to the Local Execution Environment

### Bash

You can download input data from a project using [`dx download`](https://documentation.dnanexus.com/helpstrings-of-sdk-command-line-utilities#download) in a notebook cell:

```shell
%%bash
dx download input_data/reads.fastq
```

The `%%bash` keyword converts the whole cell to a **magic cell** which allows you to run bash code in that cell without exiting the Python kernel. See examples of magic commands in the [IPython documentation](https://ipython.readthedocs.io/en/stable/interactive/magics.html). The `!` prefix achieves the same result:

```shell
! dx download input_data/reads.fastq
```

Alternatively, the `dx` command can be executed from the [terminal](#use-the-terminal).

### Python

To download data with Python in the notebook, you can use the [`download_dxfile`](http://autodoc.dnanexus.com/bindings/python/current/dxpy_dxfile.html#dxpy.bindings.dxfile_functions.download_dxfile) function:

```python
import dxpy
dxpy.download_dxfile(dxid='file-xxxx',
                     filename='unique_name.txt')
```

Check the [`dxpy` helper functions](http://autodoc.dnanexus.com/bindings/python/current/dxpy_dxfile.html#helper-functions) for details on how to download files and folders.

## Upload Data from the Session to the Project

### Bash

Any files from the execution environment can be uploaded to the project using [`dx upload`](https://documentation.dnanexus.com/helpstrings-of-sdk-command-line-utilities#upload):

```shell
%%bash
dx upload Readme.ipynb
```

### Python

To upload data using Python in the notebook, you can use the [`upload_local_file`](http://autodoc.dnanexus.com/bindings/python/current/dxpy_dxfile.html#dxpy.bindings.dxfile_functions.upload_local_file) function:

```python
import dxpy
dxpy.upload_local_file('variants.vcf')
```

Check the [`dxpy` helper functions](http://autodoc.dnanexus.com/bindings/python/current/dxpy_dxfile.html#helper-functions) for details on how to upload files and folders.

## Download and Upload Data to Your Local Machine

By selecting a notebook or any other file on your computer and dragging it into the DNAnexus project file browser, you can upload the files directly to the project. To download a file, right-click on it and click **Download (to local computer)**.

You may upload and download data to the [local execution environment](https://documentation.dnanexus.com/user/jupyter-notebooks/..#worker-execution-environment) in a similar way, that is, by dragging and dropping files to the execution file browser or by right-clicking on the files there and clicking **Download**.

## Use the Terminal

It is useful to have a terminal provided by JupyterLab at hand, which uses `bash` shell by default and lets you execute shell scripts or interact with the platform via `dx` toolkit. For example, the following command confirms what the current project context is:

```shell
$ dx pwd
MyProject:/
```

Running `pwd` shows you that the working directory of the execution environment is `/opt/notebooks`. The JupyterLab server is launched from this directory, which is also the default location of the output files generated in the notebooks.

To open a terminal window, go to **File** > **New** > **Terminal** or open it from the Launcher (using the "Terminal" box at the bottom). To open a Launcher, select **File** > **New Launcher**.

## Install Custom Packages in the Session Environment

You can install `pip`, `conda`, `apt-get`, and other packages in the execution environment from the notebook:

```shell
%%bash
pip install torch
pip install torchvision
conda install -c conda-forge opencv
```

By creating a [snapshot](https://documentation.dnanexus.com/user/jupyter-notebooks/..#environment-snapshots), you can start subsequent sessions with these packages pre-installed by providing the snapshot as input.

## Access Public and Private GitHub Repositories from the JupyterLab Terminal

You can access public GitHub repositories from the JupyterLab terminal using the `git clone` command. By placing a private ssh key that's registered with your GitHub account in `/root/.ssh/id_rsa` you can clone private GitHub repositories using `git clone` and push any changes back to GitHub using `git push` from the JupyterLab terminal.

Below is a screenshot of a JupyterLab session with a terminal displaying a script that:

* sets up an SSH key to access a private GitHub repository and clones it,
* clones a public repository,
* downloads a JSON file from the DNAnexus project,
* modifies an open-source notebook to convert the JSON file to CSV format,
* saves the modified notebook to the private GitHub repository,
* and uploads the results of JSON to CSV conversion back to the DNAnexus project.

![](https://1612471957-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-L_EsL_ie8XyZlLe_yf9%2Fuploads%2Fgit-blob-0d5e5a735a3f0aff711b341a1382832259f2fa44%2Fjl_git.png?alt=media)

This animation shows the first part of the script in action:

![](https://1612471957-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-L_EsL_ie8XyZlLe_yf9%2Fuploads%2Fgit-blob-d0c3477bea4c12999d9f5883fb6dd60242d128e5%2Fjl_github_hi_res_15s.gif?alt=media)

## Run Notebooks Non-Interactively

A command can be run in the JupyterLab Docker container without starting an interactive JupyterLab server. To do that, provide the `cmd` input and additional input files using the `in` input file array. The command runs in the directory where the JupyterLab server is started and notebooks are run, that is, `/opt/notebooks/`. Any output files generated in this directory are uploaded to the project and returned in the `out` output.

The `cmd` input makes it possible to use a `papermill` tool pre-installed in the JupyterLab environment that executes notebooks non-interactively. For example, to execute all the cells in a notebook and produce an output notebook:

```shell
my_cmd="papermill notebook.ipynb output_notebook.ipynb"
dx run dxjupyterlab -icmd="$my_cmd" -iin="notebook.ipynb"
```

where `notebook.ipynb` is the input notebook to `papermill`, which needs to be passed in the `in` input, and `output_notebook.ipynb` is the name of the output notebook, which stores the result of the cells' execution. The output is uploaded to the project at the end of the app execution.

If the `snapshot` parameter is specified, execution of `cmd` takes place in the specified Docker container. The `duration` argument is ignored when running the app with `cmd`. The app can be run from the command line with the `--extra-args` flag to limit the runtime, for example, `dx run dxjupyterlab --extra-args '{"timeoutPolicyByExecutable": {"app-xxxx":{"\*": {"hours": 1}}}}'"`.

If `cmd` is not specified, the `in` parameter is ignored and the output of an app consists of an empty array.

## Use newer NVIDIA GPU-accelerated software

If you are trying to use newer NVIDIA GPU-accelerated software, you may find that the NVIDIA GPU Driver kernel-mode driver `NVIDIA.ko` that is installed outside of the JupyterLab environment does not support the newer CUDA version required by your application. You can install [NVIDIA Forward Compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/#forward-compatibility) packages to use the newer CUDA version required by your application by following the steps below in a JupyterLab terminal.

```
# NVIDIA-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.129.06   Driver Version: 470.129.06   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
// Let's upgrade CUDA 11.4 to 12.5
# apt-get update
# apt-get -y install cuda-toolkit-12-5 cuda-compat-12-5
# echo /usr/local/cuda/compat > /etc/ld.so.conf.d/NVIDIA-compat.conf
# ldconfig
# NVIDIA-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.256.02   Driver Version: 470.256.02   CUDA Version: 12.5     |
|-------------------------------+----------------------+----------------------+
// CUDA 12.5 is now usable from terminal and notebooks
```

## Session Inactivity

After 15 to 30 minutes of inactivity in the JupyterLab browser tabs, the system logs you out automatically from the JupyterLab session and displays a "Server Connection Error" message. To re-enter the JupyterLab session, reload the JupyterLab webpage and log into the platform to be redirected to the JupyterLab session.