# Python package installation in Ubuntu 24.04 AEE

In a generic Ubuntu 24.04 environment, installing packages from the PyPI repository using `python3 -m pip install` results in an `externally-managed-environment` [error](https://packaging.python.org/en/latest/specifications/externally-managed-environments/) for both system-wide and user-specific installations (see [PEP 668](https://peps.python.org/pep-0668/)) and a recommendation to install Python packages in a Python virtual environment (`venv`).

In Ubuntu 24.04 DNAnexus Application Execution Environment (AEE), the environment variable `PIP_BREAK_SYSTEM_PACKAGES=1` is set. This environment variable disables `pip`'s externally managed environment check so that Python package installation works the same way as in Ubuntu 20.04 AEE, allowing `python3 -m pip install` outside of a Python virtual environment and simplifying migration of apps from 20.04 to 24.04 AEE.

However, this may lead to Python package version conflicts between system-wide Python packages installed via `apt-get install` and user-specific Python packages installed via `python3 -m pip install`. For example, issuing `python3 -m pip install jsonschema==4.23` in Ubuntu 24.04 AEE fails due to a conflict with the apt-installed version of `jsonschema` installed as part of the `python3-jsonschema` apt package required by the `docker-ce` apt package:

```shell
root@job-xxxx:~# aptitude why python3-jsonschema
i   docker-ce        Depends  iptables
i A iptables         Suggests firewalld
p   firewalld        Depends  python3-nftables (>= 0.9.3-2~)
p   python3-nftables Depends  python3-jsonschema

root@job-xxxx:~# python3 -m pip install jsonschema==4.23
...
Attempting uninstall: jsonschema...Found existing installation: jsonschema 4.10.3
ERROR: Cannot uninstall jsonschema 4.10.3, RECORD file not found. Hint: The package was installed by debian.
```

Such conflicts can be avoided by using Python `venv`:

```shell
root@job-xxxx:~# python3 -m venv myvenv
root@job-xxxx:~# source myvenv/bin/activate
(myvenv) root@job-xxxx:~# python3 -m pip install jsonschema==4.23
...
Successfully installed jsonschema-4.23.0 ...
```

Similar conflicts may lead to `DXExecDependencyError` when running an applet with `runSpec.execDepends` `pip3` dependencies. For example, running an applet with a `pip3` `runSpec.execDepends` that includes `ipython[all]` which depends on a Python `jsonschema` package with a version newer than 4.10.3 results in a `DXExecDependencyError` while trying to install `ipython[all]` during the job runtime:

```shell
$ grep -C2 ipython dxapp.json
{
  "package_manager": "pip3",
  "name": "ipython[all]"
}

$ dx build .
applet-xxxx

$ dx run applet-xxxx
Found existing installation: jsonschema 4.10.3
ERROR: Cannot uninstall jsonschema 4.10.3, RECORD file not found. Hint: The package was installed by debian.
…
dxpy.utils.exec_utils.DXExecDependencyError: Error while installing pip3 packages [{'name': 'ipython[all]', 'package_manager': 'pip3'}]
```

The following workarounds exist for such conflicts:

1. If an [apt package](https://packages.ubuntu.com/) with similar functionality is available, use `apt` instead of `pip3` in `runSpec.execDepends`. For example, you can install the `ipython` package (without extras) by replacing `{'name': 'ipython[all]', 'package_manager': 'pip3'}` with `{'name': 'python3-ipython'}` in the applet's `runSpec.execDepends`.
2. If the Python dependency is installed to provide a command-line tool, you can remove the conflicting `runSpec.execDepends` `pip3` dependency from `dxapp.json` and use [`pipx`](https://github.com/pypa/pipx) inside the applet script to install the command line tools
   1. Example of installing the `ipython` CLI tool using `pipx`:
      1. ```shell
         sudo apt-get install pipx -y
         pipx install ipython[all]
         pipx ensurepath
         ipython
         ```
   2. `pipx inject` can be used to install another package inside a previously installed package, e.g.
      1. `$ pipx inject ipython dxpy`
   3. `pipx` doesn't see global packages which were installed by the system
3. Otherwise, remove the conflicting `runSpec.execDepends` dependency from `dxapp.json` and use [`venv`](https://docs.python.org/3/library/venv.html) inside the applet script
   1. Example of installing the `ipython[all]` package with `venv`:
      1. ```shell
         python3 -m venv --prompt ipython ipython_venv
         source ipython_venv/bin/activate
         python3 -m pip install ipython[all]
         ```
   2. To allow `venv` to see packages installed at the system level, create your `venv` with the `--system-site-packages` flag


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://documentation.dnanexus.com/developer/apps/dependency-management/python-package-installation-in-ubuntu-24-04-aee.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.