Org Management

On the DNAnexus Platform, users can form an organization, which is an entity with a common pool of funds that all users can draw from. All projects billed to that organization (as well as all jobs that run in those projects) incur charges using the same pool of funds, regardless of user.

The All Orgs tab, which you can select from Org Admin option on the top left menu, shows you a list of all orgs to which you have admin access. On this page, you can quickly see all your orgs, the number of projects per org, and the funds available to the org. You can also see additional settings for each org by enabling optional columns (icon on the right side of the column headers).

Org Admin Guide

These operations are restricted to only org admins - that is, org members with "Admin" membership level.

As an org admin, you have access to the org admin tools from the main Platform menu. From here, you can quickly navigate to the list of orgs you administer (via "All Orgs"), or to a specific org. In this example, you would click on Demo Organization.

The "Basic Information" section of the org settings tab shows an overview of the org.

View and edit the organization name. This is how the org appears in the UI and in email notifications
View the organization ID, the unique ID used to reference a particular org on the CLI. An example is org-demo_org
View the number of org members, org projects and org apps
View the list of organization admins

Update Org Policies

You can use the dx update org command to update your org policies.

To update the membership list visibility policy so all org members can view the list of org members, use the following command, replacing org-demo with your org ID.

dx update org org-demo --member-list-visibility MEMBER

To update the restrict project transfer policy so that only org admins can change the billing account of an org project, use the following command, replacing org-demo with your org ID.

dx update org org-demo --project-transfer-ability ADMIN

Via the UI

You can also update org policies for your org in the "Policies and Administration" section of the org settings tab. Here, you can both change the membership list visibility and restrict project transfer policies for the org and contact DNAnexus Support to enable PHI data policies for org projects.

For general information about org policies, see the Organizations page.

Edit Org Membership

Within an org page, the Members tab allows you to view all the members of the org, invite new members, remove existing members, and update existing members' permission levels.

To add existing DNAnexus user to your org, use the "+ Invite New Member" button from the org's Members tab. This opens a dialog where you enter the user's username, such as smithj, or user-ID, such as user-smithj. Then configure the user's access level in the org.

Organization Membership: Whether they are a member or admin of the org.
Billable Activities Access: Whether the user should be able to create projects and apps billed to the org.
Shared Project Access: The maximum permission the user has to projects shared with the org. For example, if this is set to "Contributor" but someone shares a project with the org at an Administrator level, this user retains only Contributor access, even if other members of this org receive the full Administrator permissions.
Shared Apps Access: Whether the user should have access to apps shared with the org.

When you add a member to the org with billable activities access set to billing allowed, they gain the ability to create new projects billed to the org. However, adding the member does not change their default billing account. To use the org as their default billing account, the user must set it manually.

For any pre-existing projects not billed to the org, the user must transfer them to the org to change their billing.

The user receives an email notification after being added to the organization.

Via the CLI

The dx add member command allows org admins to add existing DNAnexus users to a pre-existing org. The command adds a member to the org but does not set their default billing account. The member must set their own default billing account manually. For any pre-existing projects not billed to the org, the user needs to transfer those projects to the org to have them billed to the org.

Below are some examples of adding users to an org.

To add an existing user to an org with:

Membership level: MEMBER
Billable activities access: Allowed
Shared apps access: Allowed
Shared projects access: CONTRIBUTE

Use the following command, replacing org-demo with your org ID and xxxx in user-xxxx with the new member's username.

# Invite user-xxxx to org-demo with CONTIRIBUTE access
dx add member org-demo user-xxxx \
  --level MEMBER --allow-billable-activities \
  --project-access CONTRIBUTE

To add an existing user to an org with:

Membership level: MEMBER
Billable activities access: Not allowed
Shared apps access: Not allowed
Shared projects access: VIEW

Use the following command, replacing org-demo with your org ID and xxxx in user-xxxx with the new member's username.

# Invite user-xxxx to org-demo with VIEW access
dx add member org-demo user-xxxx --level MEMBER --project-access VIEW --no-app-access

To add another org admin to the org, use the following command, replacing org-demo with your org ID and xxxx in user-xxxx with the new member's username.

# Invite user-xxxx to org-demo with ADMIN access
dx add member org-demo user-xxxx --level ADMIN

For more information about adding members to an org, use the dx add members command with the -h flag.

Update Membership Access

From the org Members tab, you can edit the permissions for one or multiple members of the org. The option to "Edit Access" appears when you have one or more org members selected on the Members tab.

If you select only one member, you can specify their new access via the dialog.

You can also edit multiple members simultaneously by selecting multiple members from the members list and clicking "Edit Access".

When you're editing multiple members, you have the option of changing only one access while leaving the rest as is. In the example below, we are editing the member access for User Alice and Jennifer Smith. In this case, we are updating their access so both have the ability to create new projects billed to the org, Billing Allowed. All other permissions remain the same.

Update Membership Using CLI

The dx update member command allows org admins to update a member's access within an org.

To grant an existing org member billable activities access, use the following command, replacing org-demo with your org ID and xxxx in user-xxxx with the member's username.

# Update user-xxxx to have billable activities access
dx update member org-demo user-xxxx --level MEMBER --allow-billable-activities true

To grant an existing org member shared projects access, use the following command, replacing org-demo with your org ID and xxxx in user-xxxx with the member's username.

# Update user-xxxx to have shared projects access
dx update member org-demo user-xxxx --level MEMBER --app-access true

To update an existing org member's shared projects access, use the following command, replacing org-demo with your org ID, xxxx in user-xxxx with the member's username, and ACCESS with one of ADMINISTER, CONTRIBUTE, UPLOAD, VIEW, or NONE.

# Update user-xxxx's shared projects access privileges
dx update member org-demo user-xxxx --level MEMBER --project-access ADMINISTER
Updated membership of user-xxxx in org-demo

To make an existing org member an org admin, use the following command, replacing org-demo with your org ID and xxxx in username with the member's username.

# Promote user-xxxx to org admin
dx update member org-demo user-xxxx --level ADMIN

For more information about updating members' access and level in an org, use the dx update members command with the -h flag.

Remove a Member from an Org

Org admins can remove a member from an org using the dx remove member command. This command removes the user from the org, revoking the user's access to projects and apps shared with the org. The command also, by default and if applicable, removes the user's access to apps and projects billed to the org.

From the org Members tab, you can remove one or more members from the org. The option to "Remove" appears when you have one or more org members selected on the Members tab.

This opens a confirmation dialog for removing members from the org. This action revokes the user's access to all projects and apps billed to or shared with the org.

Remove a Member from an Org using CLI

To remove a member from an org, use the following command, replacing org-demo with your org ID and xxxx user-xxxx with the member's username.

# Remove user-xxxx from org-demo
$ dx remove member org-demo user-xxxx

WARNING: About to remove user-xxxx from org-demo; project
permissions will be removed and app permissions will be removed
Please confirm [y/n]: y
Removed user-xxxx from org-demo
Removed user-xxxx from the following projects:
  None
Removed user-xxxx from the following apps:
  None

For more information about removing members from an org, use the dx remove members command with the -h flag.

Org Projects

The org's Projects tab allows you to see the list of all projects billed to the org. This list includes all projects in which you have View and above permissions as well as projects that are billed to the org in which you do not have permissions (Not a Member).

You can view all project metadata, such as the members list, data usage, and creation date. You can also view other optional columns such as project creator. To enable the optional columns, select the column from the dropdown menu to the right of the column names.

Grant Yourself Access to Org Projects

Besides viewing the list of projects, org admins can give themselves access to any project billed to the org. If you select a project in which you are not a member, you can still navigate into the project's settings page. From the project settings page, you see a button for granting yourself Administer permissions to the project.

You can also grant yourself Administer permissions when you are a member of a project billed to your org but have only View, Contribute, or Upload permissions.

List All Org Projects

To list all org projects, including projects to which you do not have access, use the dx find org projects command, replacing org-demo with your org ID.

# List all org projects
$ dx find org projects org-demo
project-xxxx : my_new_project (ADMINISTER)
project-yyyy : example_project (CONTRIBUTE)

# Return only the project IDs for all projects billed to the org
$ dx find org projects org-demo --brief
project-xxxx
project-yyyy

# Return JSON-formatted string of org projects and project metadata
dx find org projects org-demo --json

You can also use JQ (a command-line JSON parser) to format a command-delimited list of project-ID, project name, your access level , project creator, and number of bytes stored in the project.

$ dx find org projects org-demo --json \
  | jq -r '.[] .describe \
  | .id + "," + .name + "," + .level + "," + .createdBy.user + "," \
  + (.dataUsage | tostring)'
project-xxxx,my_new_project,ADMINISTER,user-xxxx,0.059
project-yyyy,example_project,CONTRIBUTE,user-yyyy,0.048

Org Billing and Finance

Accessing Org Billing Information

To see detailed billing info for an org of which you're a member, and within which you have billable activities access:

Click on your user icon in the top right corner of your screen.
Select Billing from the dropdown menu.
A list of billing accounts appears for the orgs where you're a member, and within which you have billable activities access. Information for each account is shown in a tile. This information includes the account's name and ID. For accounts with billing set up and a spending limit in place, a Remaining figure is also shown, representing the additional charges that can be incurred, before the spending limit is reached.
Find the tile with the name of the billing account for which you'd like to see detailed info. Click on the downward-facing arrow at the bottom right of the tile.
The Billing Details section shows either a prompt to set up billing, or information on billing responsibility for the org.
The Account Summary section contains detail on the billing account's spending limit and total charges incurred to date, including those for storage, compute, and egress.

The values shown in the Account Summary are lifetime values for the billing account. They do not represent outstanding charges for usage since the last contract or account settlement date.

The Remaining value, as noted above, represents the total additional charges that can be incurred, before the spending limit is reached.

The Storage and DB Cluster Charge figures are updated monthly. This update is typically done by the end of the first week of each month, with the updated figure representing storage charges incurred during the previous month.

Setting up or Updating Billing Information for an Org

To set up billing for an org, or change billing information for that org:

Open the Billing screen by following the instructions in "Access org billing information" above.
For the org for you wish to update billing information, follow the instructions in the Billing Details section to set up or update your billing information.

Request a Spending Limit Change for an Org

To request a spending limit increase for the org, contact DNAnexus Support. Include your user ID and attach a PO.

Monitoring Account Spending and Usage

Licenses are required to use the Per-Project Usage Report and Root Execution Stats Report features. Contact DNAnexus Sales for more information.

Configuration of these features, and report delivery, is handled by DNAnexus Support.

Per-Project Usage Report

This is a report issued monthly for an individual org. This report contains information on incurred compute, storage, egress, and archival costs for each project within that org, as well as any charges incurred for using public IPv4 address for workers used for compute. The report is generated as a CSV file and is delivered to a destination project selected by the org admin.

Root Execution Stats Report

A monthly org-level root execution report that lists every root execution associated with a cost. The report is generated as a CSV and is delivered to a user-specified destination project.

Learn More About An Org

List Org Members

From the org Members tab, you can quickly see the names and access levels for all org members.

To list all the members of an org using the CLI, use the dx find org members command, replacing org-demo with your org ID.

# List all org members & admin
$ dx find org members org-demo
user-xxxx : Jane Doe  (ADMIN)
user-yyyy : John Doe  (MEMBER)

# List only org admin
$ dx find org members org-demo --level ADMIN
user-xxxx : Jane Doe  (ADMIN)

# Return only user IDs of org members & admin
$ dx find org members org-demo --brief
user-xxxx
user-yyyy

List Org Apps

To list all the apps billed to your org, use the dx find org apps org-demo command, replacing org-demo with your org ID.

# Return list of all apps under a specific org
$ dx find org apps org-demo
  Example App (example_app), v0.0.1

Check Org Expiration Status

Some organizations are set to expire automatically, typically those used for testing the platform. You can check if your organization has an expiration date set.

To check the expiration status of your organization, use the dx describe command with your org ID:

dx describe org-exampleorg --json

If the response includes an expiresAt field with a future date, your organization is set to expire on that date. The value is a Unix timestamp in milliseconds. For example, "expiresAt": 1735689600000 represents January 1, 2025 at 00:00:00 UTC.

After the expiration date passes, all data related to the organization is permanently deleted.

Database data may be recoverable for a short time after expiration, but object store data is deleted permanently due to storage volume constraints. No notifications are sent to org admins about pending expiration.

Organization expiration dates are set by DNAnexus administrators for temporary organizations. To request changes to an organization's expiration date, contact DNAnexus Support.

Grant Yourself Access to Org Objects

Give Yourself Access to any Org Project

Org admins can give themselves access to any project billed to the org by using the dx invite command. To invite yourself to a project with ADMINISTER access, use the following command, replacing project-xxxx with the project ID of the org project you want access to and user-xxxx with your username. The --no-email flag suppresses the notification email about project access.

# Grant yourself access to project-xxxx
dx invite user-xxxx project-xxxx --no-email

For more information about inviting yourself to a project, use the dx invite command with the -h flag.

Grant Yourself Access to any Org App

Org admins can grant themselves developer access to any app billed to the org. App developers can publish new app versions, deprecate old app versions, and manage the group of developers and authorized users of the app.

dx add developers app-xxxx user-xxxx

Creating an Org

If you would like to set up an org for multiple DNAnexus users to join, contact us at DNAnexus Support.

Provisioning New DNAnexus Accounts

A license is required to enable org admins to provision accounts for other users. Contact DNAnexus Sales for more information.

Admins of orgs with the proper license have the ability to provision new DNAnexus accounts on behalf of the org. The user receives an email with instructions to activate their account and set their password.

For orgs with this feature enabled, create new accounts using the dx new user command.

To create a new user and add them to your org as a member with billable activities access and set their default billing account to the org, use the following command. Provide the following information:

EMAIL: the user's email -- there cannot be multiple accounts associated with the same email
USERNAME: the user's username -- must be a unique username
FIRST: the user's first name
LAST: the user's last name

$ dx new user \ 
  --username USERNAME \ 
  --email EMAIL \
  --first FIRST_NAME \
  --last LAST_NAME \
  --org org-demo \
  --level MEMBER \
  --set-bill-to
Created new user account (user-xxxx)

For more information about provisioning new user accounts, run the dx new user command with the -h flag.

Org admins can create new DNAnexus accounts on behalf of the org. Users receive an email with instructions to activate their account and set their password.

Limited Feature Note

If this feature has already been turned on for an org you administer, the option to "Create New User" appears when you invite a new member.

The dialog expands to let you specify a username, such as alice or smithj, the new user's name, and email address. This dialog automatically creates a new user account for the given email address and adds them as a member in the org.

Set the org as the default billing account by using the checkbox under the Billable Activities Access dropdown when creating a new user with Billable Activities Access set to "Billing Allowed".

Enabling PHI Data Protection for an Org

A license and a signed Business Associate Agreement are required to enable and use PHI Data Protection. Contact DNAnexus Sales for more information.

Protected Health Information, or PHI, refers to identifiable health information that can be linked to a specific person. Once enabled for a project, DNAnexus PHI Data Protection safeguards the confidentiality of data stored in that project, in compliance with HIPAA.

Data Protection Notice

A license is required to use this feature. Contact DNAnexus Sales for more information.

If the Data Protection Notice feature is enabled for a project, all users accessing the project for the first time must review and confirm their acceptance of a requirement not to egress data from the project. This feature can be enabled or disabled for a project by the project administrator. For orgs licensed to use this feature, it can be enabled or disabled for a project within the Access Policies section of the project's Settings screen.

Forwarding Job Logs To Customer's Splunk

A license is required to use this feature. For more information, contact DNAnexus Sales.

To aid in better understanding and diagnosing any issues with executions, you can have job logs forwarded to your Splunk instance. These logs contain the output of dx watch.

Enabling Forwarding for an Org

Job logs forwarding is an org-level feature. To have this feature enabled for an org, contact DNAnexus Sales.

Setting Up Forwarding

An org admin with a full-scope DNAnexus token can proceed with setup for this enabled feature by following these instructions:

Set up the Splunk HTTP Event Collector (HEC) via the Splunk web interface.

To set up forwarding, use a dx update command in the following format, substituting the URL of your HEC, your HEC authentication token, and the name of the org:

dx update org --job-logs-forwarding-json '{"url":"https://http-inputs-examplecompany.splunkcloud.com/services/collector/event","token":"abc11d22-de34-567f-g8h9-i012jk34lm34"}' org-exampleorg

Splunk logs a message on successful update of Splunk credentials with the following text:

user-xxxx, an admin of org-yyyy is enabling DNAnexus job logs forwarding with these parameters {"url": "https://http-inputs-examplecompany.splunkcloud.com/services/collector/event","tokenSignature": "<sha256OfSplunkToken>"}

Optionally, verify your org's job logs forwarding configuration, using dx describe as follows:

dx describe org-exampleorg --json --verbose | jq .jobLogsForwarding

This returns details of the configuration, in the following format:

{"jobLogsForwarding": {
  "url":"https://http-inputs-examplecompany.splunkcloud.com/services/collector/event",
  "tokenSignature": "<sha256OfSplunkToken>",
  "updated": "timestamp-in-milliseconds-from-epoch",
  "updatedBy": "user-xxxx"
}}

Disabling Forwarding

To disable job logs forwarding for an org:

# Disable job logs forwarding for org-exampleorg
dx update org --job-logs-forwarding-json '{}' org-exampleorg

When you disable forwarding, the Platform records your user ID and a timestamp in the jobLogsForwarding field for the org. View the updated configuration details with the following command:

# View the updated job logs forwarding configuration for org-exampleorg
dx describe org-exampleorg --json --verbose | jq .jobLogsForwarding

This returns details of the configuration, in the following format:

# Example output showing updated and updatedBy fields
{"jobLogsForwarding": {
  "updated": "timestamp-in-milliseconds-from-epoch",
  "updatedBy": "user-xxxx"
}}

If forwarding is already disabled for the org, the field jobLogsForwarding remains unchanged, with the API returning the code HTTP 200.

How It Works

Once the feature is set up for an org, job logs for all jobs billed to the org are forwarded to the Splunk instance via the specified Splunk HEC.

The Platform makes reasonable attempts to upload job logs to Splunk when encountering communication errors, while limiting its use of memory and CPU on the worker.

Job logs forwarding occurs outside of the application execution environment.

Org admins manage the Splunk instance's capacity to handle the volume of job logs generated by org jobs.

For cluster jobs, only job logs from the driver node are sent to Splunk.

Forwarded Job Log Format

Forwarded job log lines include the following Splunk event fields to enable efficient searching and filtering in Splunk:

jobId The id of the job generating the log message
try The try of the job generating the log message
project The project id in which the job is executing
billTo The billTo of the job generating the log message
launchedBy The user id of the job generating the log message
rootExecution The job or analysis id of the root execution of which the job is a part
executableName The name of the DNAnexus app or applet being executed by the job
entryPoint The entrypoint in the app or applet being executed by the job
level The log level of the jog log line such as EMERG, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFO, DEBUG, STDERR, STDOUT or METRICS

Detailed job metrics are forwarded as a "Detailed job metrics" job log lines that have detailed metrics values attached as event fields named after column headings in dx watch --metrics csv.

Job log lines are truncated to 8015 utf-8-encoded bytes before being forwarded to Splunk.

Job Log Limit

For orgs with this feature enabled, job logs are forwarded to Splunk without being subject to the 4 MiB limit that applies to job logs displayed by dx watch.

Errors

If an error occurs during job logs forwarding, the system retries forwarding three times. The job's description, accessible by running dx describe for the job, contains the count of job log lines that could not be forwarded after three (3) retries.

Data Egress Charges

Job logs forwarding incurs data egress charges.

Security Considerations

The customer bears responsibility for managing Splunk users' access to, and protecting sensitive/confidential information contained in, job logs and metadata that have been forwarded to the customer's Splunk instance as part of this feature. This includes responsibility for compliance with applicable data security and privacy regulations, such as the General Data Protection Regulation (GDPR).

Customer also owns all liability for maintaining the security and privacy of job logs from the point of egress from the DNAnexus Platform to storage within the Customer's system.

DNAnexus is not responsible for the accuracy of the provided HEC URL.

For more information, contact DNAnexus Sales.

Resource Usage

Customers who have this feature enabled agree to allocate a small amount of worker resources to send job logs to Splunk.

The job logs transfer from each worker to the customer's Splunk endpoint via an API request every 30 seconds, or after every 100 job log lines, whichever comes first. The system retries unsuccessful API requests three (3) times. The worker's memory buffers up to 10MiB of job logs for delivery to the Splunk endpoint. If the Splunk endpoint cannot keep up with the rate of job log production, the system drops some job log lines and records the count in the job's jobLogsForwardingStatus.linesDropped field. The additional overhead of managing the 10MiB buffer varies from 5 to 15MiB depending on the size of individual job log lines. Job log text is sent to Splunk as JSON strings, along with the metadata associated with each job log line.

Learn More

The video tutorials provide walkthroughs of key org admin tasks:

Last updated 2 days ago

Was this helpful?