# JupyterLab Reference {% hint style="info" %} JupyterLab is accessible to all users of the UK Biobank Research Analysis Platform and the Our Future Health Trusted Research Environment. A license is required to access JupyterLab on the DNAnexus Platform. [Contact DNAnexus Sales](mailto:sales@dnanexus.com) for more information. {% endhint %} ## Download Files from the Project to the Local Execution Environment ### Bash You can download input data from a project using [`dx download`](/user/helpstrings-of-sdk-command-line-utilities.md#download) in a notebook cell: ```shell %%bash dx download input_data/reads.fastq ``` The `%%bash` keyword converts the whole cell to a **magic cell** which allows you to run bash code in that cell without exiting the Python kernel. See examples of magic commands in the [IPython documentation](https://ipython.readthedocs.io/en/stable/interactive/magics.html). The `!` prefix achieves the same result: ```shell ! dx download input_data/reads.fastq ``` Alternatively, the `dx` command can be executed from the [terminal](#use-the-terminal). ### Python To download data with Python in the notebook, you can use the [`download_dxfile`](http://autodoc.dnanexus.com/bindings/python/current/dxpy_dxfile.html#dxpy.bindings.dxfile_functions.download_dxfile) function: ```python import dxpy dxpy.download_dxfile(dxid='file-xxxx', filename='unique_name.txt') ``` Check the [`dxpy` helper functions](http://autodoc.dnanexus.com/bindings/python/current/dxpy_dxfile.html#helper-functions) for details on how to download files and folders. ## Upload Data from the Session to the Project ### Bash Any files from the execution environment can be uploaded to the project using [`dx upload`](/user/helpstrings-of-sdk-command-line-utilities.md#upload): ```shell %%bash dx upload Readme.ipynb ``` ### Python To upload data using Python in the notebook, you can use the [`upload_local_file`](http://autodoc.dnanexus.com/bindings/python/current/dxpy_dxfile.html#dxpy.bindings.dxfile_functions.upload_local_file) function: ```python import dxpy dxpy.upload_local_file('variants.vcf') ``` Check the [`dxpy` helper functions](http://autodoc.dnanexus.com/bindings/python/current/dxpy_dxfile.html#helper-functions) for details on how to upload files and folders. ## Download and Upload Data to Your Local Machine By selecting a notebook or any other file on your computer and dragging it into the DNAnexus project file browser, you can upload the files directly to the project. To download a file, right-click on it and click **Download (to local computer)**. You may upload and download data to the [local execution environment](/user/jupyter-notebooks.md#worker-execution-environment) in a similar way, that is, by dragging and dropping files to the execution file browser or by right-clicking on the files there and clicking **Download**. ## Use the Terminal It is useful to have a terminal provided by JupyterLab at hand, which uses the `bash` shell by default and lets you execute shell scripts or interact with the platform via the `dx` toolkit. For example, the following command confirms what the current project context is: ```shell $ dx pwd MyProject:/ ``` Running `pwd` shows you that the working directory of the execution environment is `/opt/notebooks`. The JupyterLab server is launched from this directory, which is also the default location of the output files generated in the notebooks. To open a terminal window, go to **File** > **New** > **Terminal** or open it from the Launcher (using the "Terminal" box at the bottom). To open a Launcher, select **File** > **New Launcher**. ## Install Custom Packages in the Session Environment You can install `pip`, `conda`, `apt-get`, and other packages in the execution environment from the notebook: ```shell %%bash pip install torch pip install torchvision conda install -c conda-forge opencv ``` By creating a [snapshot](/user/jupyter-notebooks.md#environment-snapshots), you can start subsequent sessions with these packages pre-installed by providing the snapshot as input. ## Access Public and Private GitHub Repositories from the JupyterLab Terminal You can access public GitHub repositories from the JupyterLab terminal using the `git clone` command. By placing a private SSH key that's registered with your GitHub account in `/root/.ssh/id_rsa` you can clone private GitHub repositories using `git clone` and push any changes back to GitHub using `git push` from the JupyterLab terminal. Below is a screenshot of a JupyterLab session with a terminal displaying a script that: * sets up an SSH key to access a private GitHub repository and clones it, * clones a public repository, * downloads a JSON file from the DNAnexus project, * modifies an open-source notebook to convert the JSON file to CSV format, * saves the modified notebook to the private GitHub repository, * and uploads the results of JSON to CSV conversion back to the DNAnexus project. ![](/files/-M8ScGGPZQ2YDtdY2krw) This animation shows the first part of the script in action: ![](/files/-M8YO2tMMgERKYu_-PvH) ## Run Notebooks Non-Interactively A command can be run in the JupyterLab Docker container without starting an interactive JupyterLab server. To do that, provide the `cmd` input and additional input files using the `in` input file array. The command runs in the directory where the JupyterLab server is started and notebooks are run, that is, `/opt/notebooks/`. Any output files generated in this directory are uploaded to the project and returned in the `out` output. The `cmd` input makes it possible to use a `papermill` tool pre-installed in the JupyterLab environment that executes notebooks non-interactively. For example, to execute all the cells in a notebook and produce an output notebook: ```shell my_cmd="papermill notebook.ipynb output_notebook.ipynb" dx run dxjupyterlab -icmd="$my_cmd" -iin="notebook.ipynb" ``` where `notebook.ipynb` is the input notebook to `papermill`, which needs to be passed in the `in` input, and `output_notebook.ipynb` is the name of the output notebook, which stores the result of the cells' execution. The output is uploaded to the project at the end of the app execution. If the `snapshot` parameter is specified, execution of `cmd` takes place in the specified Docker container. The `duration` argument is ignored when running the app with `cmd`. The app can be run from the command line with the `--extra-args` flag to limit the runtime, for example, `dx run dxjupyterlab --extra-args '{"timeoutPolicyByExecutable": {"app-xxxx":{"\*": {"hours": 1}}}}'"`. If `cmd` is not specified, the `in` parameter is ignored and the output of an app consists of an empty array. ## Use newer NVIDIA GPU-accelerated software If you are trying to use newer NVIDIA GPU-accelerated software, you may find that the NVIDIA GPU Driver kernel-mode driver `NVIDIA.ko` that is installed outside of the JupyterLab environment does not support the newer CUDA version required by your application. You can install [NVIDIA Forward Compatibility](https://docs.NVIDIA.com/deploy/cuda-compatibility/#forward-compatibility) packages to use the newer CUDA version required by your application by following the steps below in a JupyterLab terminal. ``` # NVIDIA-smi +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.129.06 Driver Version: 470.129.06 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ // Let's upgrade CUDA 11.4 to 12.5 # apt-get update # apt-get -y install cuda-toolkit-12-5 cuda-compat-12-5 # echo /usr/local/cuda/compat > /etc/ld.so.conf.d/NVIDIA-compat.conf # ldconfig # NVIDIA-smi +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.256.02 Driver Version: 470.256.02 CUDA Version: 12.5 | |-------------------------------+----------------------+----------------------+ // CUDA 12.5 is now usable from the terminal and notebooks ``` ## Session Inactivity After 15 to 30 minutes of inactivity in the JupyterLab browser tabs, the system logs you out of the JupyterLab session automatically and displays a "Server Connection Error" message. To re-enter the JupyterLab session, reload the JupyterLab webpage and log into the platform to be redirected to the JupyterLab session. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://documentation.dnanexus.com/user/jupyter-notebooks/references.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.