Transitioning from Applets to Apps
Last updated
Was this helpful?
Last updated
Was this helpful?
Apps (applications) are packaged standalone executables that are versioned. They reside in a global app namespace, and there can be multiple versions of a single app name. Developers control the list of authorized users that can find and run the app.
Developers can build a new app version and publish it. Publishing a version makes that version available (visible) to the authorized users, and restricts further modifications to it. Users can choose to run the most recently published version, or any other previously published version.
There is no unpublish
You cannot modify published app versions. This ensures that executions of published app versions are always reproducible.
In serious cases, you might want to . For example, the published app version has a major bug and you never want users to execute it. Removed app versions are not reproducible (executions are forbidden), so be careful.
The following table summarizes the differences between applets and apps.
Where are they?
In folders inside projects.
In a global namespace. App names must be globally unique (across all customers of DNAnexus).
How are they named?
project:/folder/name
Example: dx run my-study:/applets/bwa
app-name
Example: dx run app-bwa
How are they shared?
By sharing a project, or copying an applet across projects.
The app author manages a list of users authorized to access the app.
How can they be modified?
They can be renamed or moved around folders.
They can't be modified once published.
How can they be deleted?
They can be deleted like other objects on the system, such as files, records, etc.
They can be removed.
How can they be updated?
By deleting (or renaming/moving out of the way) an existing applet and creating a new one.
By introducing a new version to an app, that is by rebuilding an app.
How are they versioned?
They are not. The system is unaware of relationships between "old" and "new" applets.
Each time an app is rebuilt, the app author must give it a new, unique version. Multiple versions of the app are available as app-name/version (i.e. dx run app-bwa/1.1). Also, app-name (with no version qualifier, i.e. dx run app-bwa) typically refers to the most recently introduced version.
Can they be "reverse-engineered" (i.e. can one retrieve the shell script and resources used in the analysis)?
Yes, with dx get.
Users can retrieve that information only if the app author has designated the app as "open-source".
Where can they store assets?
1. Inline using the resources folder during dx build.
2. In the parent project, accessible via $DX\_PROJECT\_CONTEXT\_ID from inside the script.
1. Inline using the resources folder during dx build.
2. In a private, read-only container, specific to the app version, accessible via DX\_RESOURCES\_ID.
During early development, applets are a convenient way to experiment with analyses inside own projects. Once iterations are over and applets need to be locked down (and perhaps distributed to a wider audience), transitioning to apps is an attractive option.
Once apps are published, they cannot be modified. They can only be updated by newer versions, but the system keeps all previous versions (and these are still accessible by default, unless the app author decides to remove some versions). Apps can carry their own assets in a private container (a kind of read-only project). This enhances reproducibility and minimizes risks.
Apps can be instantly shared across a list of users. In combination with being self-sufficient by storing their own assets, and being locked down, this makes it a more convenient choice for sharing them with less sophisticated users (who only need to run them).
To transition an applet into an app, you will need to address the following areas.
Think of a name for your app. Certain app names, such as bwa or fastqc, are already taken.
To maintain clarity and avoid naming conflicts, consider using a unique prefix for your applications. For example, if you're developing tools within the Center for Cancer Informatics, you might name them "cci-bwa", "cci-fastqc," and so on. This would result in commands like dx run app-cci-bwa/1.2, where "cci-" identifies the application's origin.
Think of an initial version number for your app. Choosing a versioning convention is very important for you to perform meaningful app updates later.
Decide if your app is open-source. Do you want to let users retrieve the shell script and any other resources associated with the app?
In dxapp.json, add a key called openSource with a boolean value of true or false.
Decide the app author. Just like applets, apps require storage for their resources and assets, and for that storage they will be billed on a monthly basis to the billing account of the original author of each app.
Additional authors, that is users who can update the app by building a newer version, can be added or removed at any time, but it is the billing account of the original author that will always be associated with billing of storage for this app.
Do not author apps under a trial account. Whoever you decide to be the original app author, ask them to run the very first dx build of the app.
Decide which users will be allowed to access your app. The app author always has access to the app.
If you intend to run the apps under the same user as the one authoring them, you do not need to do anything. Otherwise, in dxapp.json, add a key called authorizedUsers with a value being an array of strings, corresponding to the entities allowed to run the app. Entities are encoded as user-username for single users, or org-orgname for organizations (i.e. "authorizedUsers": ["user-george", "org-cci"]). You can also manage the list of authorized users at any point via dx list users, and dx remove users.
If you prefer to use dx, then omit the "authorizedUsers" key from dxapp.json altogether. (The list of authorized users is common to all versions of the app).
If the app requires assets, that is files that reside in a project and not inside the ./resources/ folder at dx build time, you need to perform the following changes.
In the bash script, when fetching assets, replace $DX_PROJECT_CONTEXT_ID with $DX_RESOURCES_ID, so that your script fetches the assets from the app's private container and not from inside whatever parent project it may be run. NOTE: If you want for the same script to be able to function both as an applet and as an app, then add code like the following (which introduces a new variable), and use $DX_ASSETS_ID when fetching assets:
Create a project with the exact structure that you want your assets to have, i.e. for the example above, create a project with a folder "/assets/" and place hs37d5.fasta inside that folder.
In dxapp.json, add a key called resources with a string value equal to the project ID of the project with the assets, i.e. "resources": "project-BBF4Jp80vVky55Vvgkb0028u"
When building the app, the system will create a separate read-only copy of the project and associate it with the specific app version. You can safely modify or delete the original project, as it will not affect the app. Note that if your assets don't change across newer versions, you are welcome to keep the original project around and reuse the same project ID in dxapp.json when rebuilding the app later; but the system will still create a separate read-only copy for each version.
Unlike an applet, an app can be run in multiple regions, which are denoted by the enabled regions of the app. For a given app version, the set of enabled regions is immutable.
To build the app, run the build command:
The first time you run this command, the system will reserve the app-name in the global namespace, and create the first version of the app according to the version key in dxapp.json.
You can still build this as an applet, by omitting the --app and --publish options.
To perform subsequent updates, after you make any changes to the code, increment the version in dxapp.json, and rerun the dx build --app --publish command.
In the examples above, you will notice that new app versions are built and published at the same time, by supplying the --publish option to dx build. Publishing an app version makes it available to authorized users and ensures that it can no longer be modified.
DNAnexus suggests following that workflow, but it is not a requirement.
You can build an app version without publishing it by omitting the --publish option. The generated app version will only be accessible by the app developers, that is the original app author and any users added via dx add developers, and not by authorized users. You can use this flow if you want to first test the new app version before you publish it.
During testing, if you find that you must make further changes, you can update the same unpublished app version in-place, without having to generate a new version. Use dx build --app again, and if the version in dxapp.json matches an unpublished existing version, that version will be overwritten. Once you are happy with the app version, you can publish it by repeating the dx build command with the --publish argument.
Similarly to applets, each app version receives a unique id of the form app-xxxx. It is also accessible via the "app-name/version" scheme, for example, app-cci-bwa/1.0.0, which cannot be changed once that app version is published.
Apps have the concept of a default version. If an app has multiple published versions, there is exactly one version at all times which is the default.
When you run dx build --app --publish, the published app version becomes the new default. Typically, the default version is whatever version was most recently published.
Users refer to the default version when using "app-name" with no version qualifier. For example, when executing dx run app-cci-bwa. Users who launch apps have a choice of whether to use the default version (by doing dx run app-cci-bwa) or to use a specific version (by doing dx run app-cci-bwa/1.0.0).
If you publish a new app version, such as app-cci-bwa/1.0.1, and later realize that you've made a mistake that needs fixing, you have two options:
Create a new app version, such as app-cci-bwa/1.0.2, that doesn't have the problem and publish it. Publishing will make 1.0.2 the new default version. As a result, users who execute dx run app-cci-bwa will run a working version.
Mark a previous app version, such as app-cci-bwa/1.0.0 as the default version. You can do by tagging that version as default using dx api app-cci-bwa/1.0.0 addTags '{"tags":["default"]}'.
Apps cannot be modified once published. However, you can remove a specific app version if necessary. For example, an app version might have a critical flaw or bug and you never want users to execute it. In such cases, you can remove an app version so it's no longer allowed to run.
For removed app versions, users can still get basic information about the version using dx describe. However, the input/output spec, actual code, and assets of the removed app version aren't available.
DNAnexus strongly suggests the conventions. Under those conventions, your first production version should be "1.0.0". In dxapp.json, add a key called "version" with a string value equal to the version you chose ("1.0.0"). For a trivial update (such as a bugfix), you should later increase it to "1.0.1". For a minor update which is backwards compatible (such as updating the underlying software version of bwa), you should increase it to "1.1.0". For a major upgrade such as a backwards-incompatible change in the input/output spec, you should increase it to "2.0.0", etc.
To set the enabled regions for an app, specify the regionalOptions key in dxapp.json, according to the .
To remove an app version, that version must not be the default. If you need to remove the current default version of an app, you first need to .