You can download input data from a project using dx download
in a notebook cell:
%%bashdx download input_data/reads.fastq
The %%bash
keyword converts the whole cell to a magic cell which allows us to run bash code in that cell without exiting the Python kernel. See me examples of magic commands in the IPython documentation. The !
prefix to achieves the same result:
! dx download input_data/reads.fastq
Alternatively, the dx
command can be executed from the terminal.
To download data with Python in the notebook, you can use the download_dxfile
function:
import dxpydxpy.download_dxfile(dxid='file-xxxx',filename='unique_name.txt')
Check dxpy helper functions for details on how to download files and folders.
Any files from the execution environment can be uploaded to the project using dx upload
:
%%bashdx upload Readme.ipynb
To upload data using Python in the notebook, you can use the upload_local_file
function:
import dxpydxpy.upload_local_file('variants.vcf')
Check dxpy helper functions for details on how to upload files and folders.
By selecting a notebook or any other file on your computer and dragging it into the DNAnexus project file browser, you can upload the files directly to the project. To download a file, right-click on it and click Download (to local computer)
.
You may upload and download data to the local execution environment in a similar way, i.e. by dragging and dropping files to the execution file browser or by right-clicking on the files there and clicking Download
.
It is useful to have a terminal provided by JupyterLab at hand, which uses bash
shell by default and lets you execute shell scripts or interact with the platform via dx
toolkit. For example, the command:
$ dx pwdMyProject:/
will confirm what the current project context is.
Running pwd
will show you that the working directory of the execution environment is /opt/notebooks
. The JupyterLab server is launched from this directory, which is also the default location of the output files generated in the notebooks.
To open a terminal window, go to File
> New
> Terminal
or open it from the Launcher (using the "Terminal" box at the bottom). To open a Launcher, select File
> New Launcher
.
You can install pip
, conda
, apt-get
, and other packages in the execution environment from the notebook:
%%bashpip install torchpip install torchvisionconda install -c conda-forge opencv
By creating a snapshot, you can start subsequent sessions with these packages pre-installed by providing the snapshot as input.
You can access public github repositories from the JupyterLab terminal using git clone
command. By placing a private ssh key that's registered with your github account in /root/.ssh/id_rsa,
you can clone private github repositories using git clone
and push any changes back to github using git push
from the JupyterLab terminal.
Below is a screenshot of a JupyterLab session with a terminal displaying a script that:
sets up ssh key to access a private github repository and clones it,
clones a public repository,
downloads a json file from the DNAnexus project,
modifies an open-source notebook to convert the json file to csv format,
saves the modified notebook to the private github repository,
and uploads the results of json to csv conversion back to the DNAnexus project.
This animation shows the first part of the script in action: