Python package installation in Ubuntu 24.04 AEE

In a generic Ubuntu 24.04 environment, installing packages from the PyPI repository using python3 -m pip install results in an externally-managed-environment error for both system-wide and user-specific installations (see PEP 668) and a recommendation to install Python packages in a Python virtual environment (venv).

In Ubuntu 24.04 DNAnexus Application Execution Environment (AEE), the environment variable PIP_BREAK_SYSTEM_PACKAGES=1 is set. This environment variable disables pip's externally managed environment check so that Python package installation works the same way as in Ubuntu 20.04 AEE, allowing python3 -m pip install outside of a Python virtual environment and simplifying migration of apps from 20.04 to 24.04 AEE.

However, this may lead to Python package version conflicts between system-wide Python packages installed via apt-get install and user-specific Python packages installed via python3 -m pip install. For example, issuing python3 -m pip install jsonschema==4.23 in Ubuntu 24.04 AEE will fail due to a conflict with apt-installed version of jsonschema installed as a part of the python3-jsonschema apt package required by docker-ce apt package:

root@job-xxxx:~# aptitude why python3-jsonschema
i   docker-ce        Depends  iptables
i A iptables         Suggests firewalld
p   firewalld        Depends  python3-nftables (>= 0.9.3-2~)
p   python3-nftables Depends  python3-jsonschema

root@job-xxxx:~# python3 -m pip install jsonschema==4.23
...
Attempting uninstall: jsonschema...Found existing installation: jsonschema 4.10.3
ERROR: Cannot uninstall jsonschema 4.10.3, RECORD file not found. Hint: The package was installed by debian.

Such conflict can be avoided by using Python venv:

root@job-xxxx:~# python3 -m venv myvenv
root@job-xxxx:~# source myvenv/bin/activate
(myvenv) root@job-xxxx:~# python3 -m pip install jsonschema==4.23
...
Successfully installed jsonschema-4.23.0 ...

Similar conflicts may lead to DXExecDependencyError when running an applet with runSpec.execDepends pip3 dependencies. For example, running an applet with a pip3 runSpec.execDepends that includes ipython[all] which depends on a Python jsonschema package with a version newer than 4.10.3 will result in a DXExecDependencyError while trying to install ipython[all] during the jobs's runtime:

$ grep -C2 ipython dxapp.json
{
  "package_manager": "pip3",
  "name": "ipython[all]"
}

$ dx build .
applet-xxxx

$ dx run applet-xxxx
Found existing installation: jsonschema 4.10.3
ERROR: Cannot uninstall jsonschema 4.10.3, RECORD file not found. Hint: The package was installed by debian.
…
dxpy.utils.exec_utils.DXExecDependencyError: Error while installing pip3 packages [{'name': 'ipython[all]', 'package_manager': 'pip3'}]

There are a few workarounds for such conflicts:

1. If there is an apt package with similar functionality, use apt instead of pip3 in runSpec.execDepends. For example, you can install ipython package (without extras) by replacing {'name': 'ipython[all]', 'package_manager': 'pip3'} with {'name': python3-ipython'} in applet's runSpec.execDepends.

2. If the Python dependency is installed in order to provide a command-line tool, you can remove the conflicting runSpec.execDepends pip3 dependency from dxapp.json and use pipx inside the applet script to install the command line tools

   1. Example of installing ipython CLI tool using pipx:

      1. sudo apt-get install pipx -y
         pipx install ipython[all]
         pipx ensurepath
         ipython

   2. pipx inject can be used to install another package inside a previously installed package, e.g.

      1. $ pipx inject ipython dxpy

   3. pipx doesn't see global packages which were installed by system

3. Otherwise, remove the conflicting runSpec.execDepends dependency from dxapp.json and use venv inside the applet script

   1. Example of installing ipython[all] package with venv:

      1. python3 -m venv --prompt ipython ipython_venv
         source ipython_venv/bin/activate
         python3 -m pip install ipython[all]

   2. To allow venv to see packages installed at the system level, create your venv with the --system-site-packages flag