Transitioning from Applets to Apps
Introduction to Apps
Apps (applications) are packaged standalone executables that are versioned. They reside in a global app namespace, and multiple versions can be created for a single app name. Developers control the list of authorized users that can find and run the app.
Developers can build a new app version, and publish it. Publishing a version makes that version available (visible) to the authorized users, and restricts further modifications to it. Users can choose to run the most recently published version, or any other previously published version.
NOTE: There is no way to unpublish or otherwise modify a published app version; this ensures that executions of published app versions are always reproducible. There is, however, a way to deprecate a published app version, which forbids all future executions. This app version deprecation is meant to be used only in serious cases where a previously published app version has serious bugs and you never want users to run it. Deprecated app versions are not reproducible, so be careful when using that feature.
Differences Between Applets and Apps
The following table summarizes the differences between applets and apps.
Applets
Apps
Where are they?
In folders inside projects.
In a global namespace. App names must be globally unique (across all customers of DNAnexus).
How are they named?
project:/folder/name (i.e. dx run my-study:/applets/bwa)
app-name (i.e. dx run app-bwa). See below for versioning.
How are they shared?
By sharing a project, or copying an applet across projects.
The app author manages a list of users authorized to access the app.
How can they be modified?
They can be renamed or moved around folders.
They can't be modified once published.
How can they be deleted?
They can be deleted like other objects on the system (files, etc)
They can be "deprecated".
How can they be updated?
By deleting (or renaming/moving out of the way) an existing applet and creating a new one.
By introducing a new version to an app (i.e. by rebuilding an app).
How are they versioned?
They are not. The system is unaware of relationships between "old" and "new" applets.
Each time an app is rebuilt, the app author must give it a new, unique version. Multiple versions of the app are available as app-name/version (i.e. dx run app-bwa/1.1). Also, app-name (with no version qualifier, i.e. dx run app-bwa) typically refers to the most recently introduced version.
Can they be "reverse-engineered" (i.e. can one retrieve the shell script and resources used in the analysis)?
Yes, with dx get.
Users can retrieve that information only if the app author has designated the app as "open-source".
Where can they store assets?
1. Inline using the "resources" folder during dx build. 2. In the parent project, accessible via $DX\_PROJECT\_CONTEXT\_ID from inside the script.
1. Inline using the "resources" folder during dx build. 2. In a private, read-only container, specific to the app version, accessible via DX\_RESOURCES\_ID.
Why Transition From Applets to Apps?
During early development, applets are a convenient way to experiment with analyses inside own projects. Once iterations are over and applets need to be locked down (and perhaps disseminated to a wider audience), transitioning to apps is an attractive option:
Once published, apps cannot be modified. They can only be updated by newer versions, but the system keeps all previous versions (and these are still accessible by default, unless the app author decides to deprecate some versions). Apps can carry their own assets in a private container (a kind of read-only project). This enhances reproducibility and minimizes risks.
Apps can be instantly shared across a list of users. In combination with being self-sufficient by storing their own assets, and being locked down, this makes it a more convenient choice for sharing them with less sophisticated users (who only need to run them).
Making the Transition
To transition an applet into an app, you will need to address these things:
App Name
Think of a name for your app. Since some app names (for example: "bwa", "fastqc", etc.) are already taken, avoid polluting the global namespace further, by introducing an additional prefix of your own as a naming convention; for example, the fictitious "Center for Cancer Informatics" could name its apps "cci-bwa", "cci-fastqc", etc. The invocation would then look like dx run app-cci-bwa/1.2, etc. where the command line includes app- before the app name. You can edit the name in dxapp.json.
App Versioning
Think of an initial version number for your app. Choosing a versioning convention is very important for you to perform meaningful app updates later.
DNAnexus strongly suggests the Semantic Versioning conventions. Under those conventions, your first production version should be "1.0.0". In dxapp.json, add a key called "version" with a string value equal to the version you chose ("1.0.0"). For a trivial update (such as a bugfix), you should later increase it to "1.0.1". For a minor update which is backwards compatible (such as updating the underlying software version of bwa), you should increase it to "1.1.0". For a major upgrade such as a backwards-incompatible change in the input/output spec, you should increase it to "2.0.0", etc.
Open Source
Decide if your app will be open-source, i.e. if you want users to be able to retrieve the shell script and any other resources associated with the app. In dxapp.json, add a key called "openSource" with a boolean value of true or false.
App Developers
Decide who is the app author. Just like applets, apps require storage for their resources and assets, and for that storage they will be billed on a monthly basis to the billing account of whomever is the original author of each app. Additional authors (users who can update the app by rebuilding a newer version) can be added or removed at any time, but it is the billing account of the original author that will always be associated with billing of any storage for this app.
Do not author apps under a trial account. Whoever you decide to be the original app author, ask them to run the very first dx build of the app.
Authorized Users
Decide which users will be allowed to access your app. The app author always has access to the app. If you intend to run the apps under the same user as the one authoring them, you do not need to do anything. Otherwise, in dxapp.json, add a key called "authorizedUsers" with a value being an array of strings, corresponding to the entities allowed to run the app. Entities are encoded as user-username for single users, or org-orgname for organizations (i.e. "authorizedUsers": ["user-george", "org-cci"]). You can also manage the list of authorized users at any point via dx list users, and dx remove users.
If you prefer to use dx, then omit the "authorizedUsers" key from dxapp.json altogether. (The list of authorized users is common to all versions of the app).
App Assets
If the app requires assets, i.e. files that reside in a project (and not inside the "./resources/" folder at dx build time), perform the following changes:
In the bash script, when fetching assets, replace $DX_PROJECT_CONTEXT_ID with $DX_RESOURCES_ID, so that your script fetches the assets from the app's private container and not from inside whatever parent project it may be run. NOTE: If you want for the same script to be able to function both as an applet and as an app, then add code like the following (which introduces a new variable), and use $DX_ASSETS_ID when fetching assets:
Create a project with the exact structure that you want your assets to have, i.e. for the example above, create a project with a folder "/assets/" and place hs37d5.fasta inside that folder.
In dxapp.json, add a key called "resources" with a string value equal to the project id of the project with the assets, i.e. "resources": "project-BBF4Jp80vVky55Vvgkb0028u"
When building the app, the system will create a separate read-only copy of the project and associate it with the specific app version. You can safely modify or delete the original project, as it will not affect the app. Note that if your assets don't change across newer versions, you are welcome to keep the original project around and reuse the same project ID in dxapp.json when rebuilding the app later; but the system will still create a separate read-only copy for each version.
Enabled Regions
Unlike an applet, an app can be run in multiple regions, which are denoted by the enabled regions of the app. For a given app version, the set of enabled regions is immutable. To specify the enabled regions of an app, please specify the regionalOptions key in dxapp.json, according to the dxapp.json specification.
Building the App
To build the app, run the following command:
The first time you run this command, the system will reserve the app-name in the global namespace, and create the first ever version of the app according to the "version" key in dxapp.json. Note also that you can still build this as an applet, by omitting the --app and --publish options.
Updating the App
To perform subsequent updates, after make any changes to the code, increment the version in dxapp.json, and rerun the aforementioned dx build --app --publish command.
Publishing an App
In the examples above, you will notice that new app versions are built and published at the same time, by supplying the --publish option to dx build. Publishing an app version makes it available to authorized users and ensures that it can no longer be modified.
DNAnexus suggests following that workflow, but it is not a requirement. You can build an app version without publishing it; simply omit the --publish option. In that case, the generated app version will only be accessible by the app developers (that is, the original app author and any users added via dx add developers), and not by authorized users. You can use that flow if you want to first test the new app version before you publish it. During testing, if you find that you must make further changes, you can update the same unpublished app version in-place, without having to generate a new version. Simply use dx build --app again, and if the version in dxapp.json matches an unpublished existing version, that version will be overwritten. Once you are happy with the app version, you can publish it by repeating the dx build command with the --publish argument.
Accessing Different App Versions
Similarly to applets, each app version receives a unique id of the form app-xxxx. It is also accessible via the "app-name/version" scheme, i.e. app-cci-bwa/1.0.0, which cannot be changed once that app version is published.
Moreover, apps have the concept of a default version. If an app has multiple published versions, there is exactly one version at all times which is the default. When you run dx build --app --publish, the published app version becomes the new default. Therefore, typically the default version is whatever version was most recently published. Users refer to the default version when using "app-name" with no version qualifier, i.e. when doing dx run app-cci-bwa. Therefore, users who launch apps have a choice of whether to use the default version (by doing dx run app-cci-bwa) or to use a specific version (by doing dx run app-cci-bwa/1.0.0).
Inside workflows, apps are always versioned; in other words, a workflow always refers to a particular app version (whichever app version was added to the workflow at the time of workflow creation). When editing the workflow in the DNAnexus web interface, if the default app version is different than the app version in the workflow (which is an indication of the app having been updated in the meantime), the website will prompt the user to optionally update the workflow to the new version.
Deprecating an App Version
If you have published a new app version (say, app-cci-bwa/1.0.1), and later realize that you have made a mistake and want to "roll back" the update, you have two options:
You can create an even newer app version (say,
app-cci-bwa/1.0.2) that does not have the problem and publish that. Publishing will make it the new default, so those who do "dx run app-cci-bwa" will run a working version.Mark a previous version (say, app-cci-bwa/1.0.0) as the default. You can do that with the following command:
In addition, you can deprecate an app version. As mentioned in the introduction, apps cannot be modified once published, but the system offers a feature to deprecate an app version. Deprecation of an app version means that the particular version will no longer be allowed to run. You will still be able to get some basic information for any deleted app version via dx describe, but the input/output spec, actual code and asset project of the deprecated app version will no longer be available. (See also the last paragraph of the introduction).
To deprecate an app version, that version must not be the default. If you need to deprecate the default version of an app, you must first either publish another version and make it default, or mark some other version as the default (as discussed above).
You can deprecate an app version with the following command:
Last updated