Org Management
On the DNAnexus Platform, several users can form an organization, which is an entity with a common pool of funds that all users can draw from. All projects billed to that organization (as well as all jobs that are run in those projects) will incur charges using the same pool of funds, regardless of user.
The All Orgs tab, which you can select from Org Admin option on the top left menu, shows you a list of all orgs to which you have admin access. On this page, you can quickly see all of your orgs, the number of projects per org, and the funds available to the org. You can also see additional settings for each org by enabling optional columns (icon on the right side of the column headers).
Org Admin Guide
These operations are restricted to only org admins - that is, org members with "Admin" membership level.
If you are an admin of an org, you will be able to access the org admin tools from the main Platform menu. From here, you can quickly navigate to the list of orgs you administer (via "All Orgs"), or to a specific org (e.g. Demo Organization in this example).
In the "Basic Information" section of the org settings tab, you will be able to see an overview of the org.
View and edit the organization name. This is how the org will be referred to in the UI and in email notifications
View the organization ID, the unique ID used to reference a particular org on the CLI (e.g. org-demo_org)
View the number of org members, org projects and org apps
View the list of organization admins
Update Org Policies
You can use the dx update org command to update your org policies.
To update the membership list visibility policy so all org members can view the list of org members, use the following command, replacing org-demo with your org ID.
To update the restrict project transfer policy so that only org admins can change the billing account of an org project, use the following command, replacing org-demo with your org ID.
Via the UI
You can also update org policies for your org in the "Policies and Administration" section of the org settings tab. Here, you can both change the membership list visibility and restrict project transfer policies for the org and contact DNAnexus support to enable PHI data policies for org projects.
For general information about org policies, please see the Organizations page.
Edit Org Membership
Within an org page, the Members tab allows you to view all the members of the org, invite new members, remove existing members, and update existing members' permission levels.
To add existing DNAnexus user to your org, you can use the "+ Invite New Member" button from the org's Members tab. This will open a dialog where you can enter the user's username (e.g. smithj) or user-ID (e.g. user-smithj). Then you can configure the user's access level in the org.
Organization Membership: Whether they are a member or admin of the org.
Billable Activities Access: Whether the user should be able to create projects and apps billed to the org.
Shared Project Access: The maximum permission the user will have to projects that were shared with the org. For example, if this is set to "Contributor" but someone shares a project with the org at an Administrator level, this user will still only have Contributor access, even if other members of this org receive the full Administrator permissions.
Shared Apps Access: Whether the user should have access to apps shared with the org.
If you add a member to the org with billable activities access set to billing allowed, they will have the ability to create new projects billed to the org. However, adding the member will not change their default billing account. If the user wishes to use the org as their default billing account, they will have to set their own default billing account.
Additionally, if the member has any pre-existing projects that are not billed to the org, the user will need to transfer the project to the org if they wish to have the project billed to the org.
The user will receive an email notification informing them that they have been added to the organization.
Via the CLI
The dx add member command allows org admins to add existing DNAnexus users to a pre-existing org. The command will add a member to the org but will not set their default billing account; the member will have to set their own default billing account. Additionally, if the member has any pre-existing projects that are not billed to the org, the user will need to transfer the project to the org if they wish to have the project billed to the org.
Below are some examples of adding users to an org.
To add an existing user to an org with:
Membership level: MEMBER
Billable activities access: Allowed
Shared apps access: Allowed
Shared projects access: CONTRIBUTE
Use the following command, replacing org-demo with your org ID and xxxx in user-xxxx with the new member's username.
To add an existing user to an org with:
Membership level: MEMBER
Billable activities access: Not allowed
Shared apps access: Not allowed
Shared projects access: VIEW
Use the following command, replacing org-demo with your org ID and xxxx in user-xxxx with the new member's username.
To add another org admin to the org, use the following command, replacing org-demo with your org ID and xxxx in user-xxxx with the new member's username.
For more information about adding members to an org, use the dx add members command with the -h flag.
Update Membership Access
From the org Members tab, you can edit the permissions for one or multiple members of the org. The option to "Edit Access" appears when you have one or more org members selected on the Members tab.
If you select only one member, you can specify their new access via the dialog.
You can also edit multiple members at the same time by selecting multiple members from the members list and clicking "Edit Access".
When you're editing multiple members, you have the option of changing only one access while leaving the rest as is. In the example below, we are editing the member access for User Alice and Jennifer Smith. In this case, we updating their access so both have the ability to create new projects billed to the org, Billing Allowed. We will keep all the rest of their permissions the same.
Via the CLI
The dx update member command allows org admins to update a member's access within an org.
To grant an existing org member billable activities access, use the following command, replacing org-demo with your org ID and xxxx in user-xxxx with the member's username.
To grant an existing org member shared projects access, use the following command, replacing org-demo with your org ID and xxxx in user-xxxx with the member's username.
To update an existing org member's shared projects access, use the following command, replacing org-demo with your org ID, xxxx in user-xxxx with the member's username, and ACCESS with one of ADMINISTER, CONTRIBUTE, UPLOAD, VIEW, or NONE.
To make an existing org member an org admin, use the following command, replacing org-demo with your org ID and xxxx in username with the member's username.
For more information about updating members' access and level in an org, use the dx update members command with the -h flag.
Remove a Member from an Org
Org admins can remove a member from an org using the dx remove member command. This command will remove the user from the org, revoking the user's access to projects and apps shared with the org. This command will also, by default and if applicable, remove the user's access to apps and projects billed to the org.
From the org Members tab, you can remove one or more members from the org. The option to "Remove" appears when you have one or more org members selected on the Members tab.
This will open the following dialog where you can confirm the action to remove members from the org. This action will revoke the user's access to all projects and apps billed to or shared with the org.
Via the CLI
To remove a member from an org, use the following command, replacing org-demo with your org ID and xxxx user-xxxx with the member's username. You will be prompted to confirm the removal of the user from the org.
For more information about removing members from an org, use the dx remove members command with the -h flag.
Org Projects
The org's Projects tab allows you to see the list of all projects billed to the org. This list will include all projects in which you have View and above permissions as well as projects that are billed to the org in which you do not have permissions (Not a Member).
You will be able to view all project metadata (e.g. the members list, data usage, creation date), as well as some other optional columns (e.g. project creator). To enable the optional columns, select the column from the dropdown menu to the right of the column names.
Grant Yourself Access to Org Projects
In addition to viewing the list of projects, org admins can give themselves access to any project billed to the org. If you select a project in which you are not a member, you will still be able to navigate into the project's settings page. From the project settings page, you will see a button allowing you to grant yourself Administer permissions to the project.
You will also be able to grant yourself Administer permissions if you are currently a member of a project billed to your org but you only have View, Contribute, or Upload permissions.
List All Org Projects
To list all org projects, including projects to which you do not have access, use the dx find org projects command, replacing org-demo with your org ID.
Org Billing and Finance
Accessing Org Billing Information
To see detailed billing info for an org of which you’re a member, and within which you have billable activities access:
Click on your user icon in the top right corner of your screen.
Select Billing from the dropdown menu.
You'll see a list of billing accounts for orgs of which you're a member, and within which you have billable activities access. Information for each account is shown in a tile. This information includes the account's name and ID. For accounts with billing set up and a spending limit in place, a Remaining figure is also shown, representing the additional charges that can be incurred, before the spending limit is reached.
Find the tile with the name of the billing account for which you'd like to see detailed info. Click on the downward-facing arrow at the bottom right of the tile.
In the Billing Details section, you'll see either a prompt to set up billing, or information on billing responsibility for the org.
In the Account Summary section, you'll see detail on the billing account's spending limit and total charges incurred to date, including those for storage, compute, and egress.
The values shown in the Account Summary are lifetime values for the billing account. They do not represent outstanding charges for usage since the last contract or account settlement date.
The Remaining value, as noted above, represents the total additional charges that can be incurred, before the spending limit is reached.
The Storage and DB Cluster Charge figures are updated monthly. Note that this update is typically done by the end of the first week of each month, with the updated figure representing storage charges incurred during the previous month.
Setting up or Updating Billing Information for an Org
To set up billing for an org, or change billing information for that org:
Open the Billing screen by following the instructions in “Access org billing information” above.
For the org for you wish to update billing information, follow the instructions in the Billing Details section to set up or update your billing information.
Request a Spending Limit Change for an Org
As an org admin, you can request a spending limit increase for the org by contacting DNAnexus Support at support@dnanexus.com. Please include your user ID and attach a PO.
Monitoring Account Spending and Usage
Licenses are required to use the Per-Project Usage Report and Root Execution Stats Report features. Contact DNAnexus Sales for more information.
Configuration of these features, and report delivery, is handled by DNAnexus Support.
Per-Project Usage Report
This is a report issued monthly for an individual org. This report contains information on incurred compute, storage, egress, and archival costs for each project within that org, as well as any charges incurred for using public IPv4 address for workers used for compute. The report is generated as a CSV file and is delivered to a destination project selected by the org admin.
Root Execution Stats Report
A monthly org-level root execution report that lists every root execution associated with a cost. The report is generated as a CSV and is delivered to a user-specified destination project.
Learn More About An Org
List Org Members
From the org Members tab, you can quickly see the names and access levels for all org members.
To list all the members of an org using the CLI, use the dx find org members command, replacing org-demo with your org ID.
List Org Apps
To list all the apps billed to your app, use the dx find org apps command, replacing org-demo with your org ID.
Grant Yourself Access to Org Objects
Give Yourself Access to any Org Project
Org admins can give themselves access to any project billed to the org by using the dx invite command. To invite yourself to a project with ADMINISTER access, use the following command, replacing project-xxxx with the project ID of the org project you want access to and user-xxxx with your username. The --no-email flag will suppress the email that would normally be sent notifying you that you have been given access to a project.
For more information about inviting yourself to a project, use the dx invite command with the -h flag.
Grant Yourself Access to any Org App
Org admins can grant themselves developer access to any app billed to the org. App developers can publish new app versions, deprecate old app versions, and manage the group of developers and authorized users of the app.
Creating an Org
If you would like to set up an org for several DNAnexus users to join, please contact us at DNAnexus support.
Provisioning New DNAnexus Accounts
A license is required to enable org admins to provision accounts for other users. Contact DNAnexus Sales for more information.
Admins of orgs with the proper license have the ability to provision new DNAnexus accounts on behalf of the org. The user will then receive an email with instructions to activate their account and set their password.
If this feature has already been turned on for an org you administer, you can create new accounts using the dx new user command.
To create a new user and add them to your org as a member with billable activities access and set their default billing account to the org, use the following command. You will need to provide the following information:
EMAIL: the user's email -- there cannot be multiple accounts associated with the same emailUSERNAME: the user's username -- must be a unique usernameFIRST: the user's first nameLAST: the user's last name
For more information about provisioning new user accounts, run the dx new user command with the -h flag.
Org admins have the ability to create new DNAnexus accounts on behalf of the org. The user will then receive an email with instructions to activate their account and set their password.
Limited Feature Note
If this feature has already been turned on for an org you administer, you will see an option to "Create New User" when you invite a new member.
The dialog will expand to allow you to specify a username (e.g. alice or smithj), the new user's name, and email address. This dialog will automatically create a new user account for the given email address and add them as a member in the org.
Please note that if you create a new user and set their Billable Activities Access to "Billing Allowed", we recommend that you set the org as the user's default billing account. This option is available as a checkbox under the Billable Activities Access dropdown.
Enabling PHI Data Protection for an Org
A license and a signed Business Associate Agreement are required to enable and use PHI Data Protection. Contact DNAnexus Sales for more information.
Protected Health Information, or PHI, refers to identifiable health information that can be linked to a specific person. Once enabled for a project, DNAnexus PHI Data Protection safeguards the confidentiality of data stored in that project, in compliance with HIPAA.
For more on PHI data protection, consult the PHI Data Protection section.
Data Protection Notice
A license is required to use this feature. Contact DNAnexus Sales for more information.
If the Data Protection Notice feature is enabled for a project, all users accessing the project for the first time will be required to review and confirm their acceptance of a requirement not to egress data from the project. This feature can be enabled or disabled for a project by the project administrator. For orgs licensed to use this feature, it can be enabled or disabled for a project within the Access Policies section of the project's Settings screen.
Forwarding Job Logs To Customer's Splunk
A license is required to use this feature. For more information, contact DNAnexus Sales.
To aid in better understanding and diagnosing any issues with executions, you can have job logs (i.e. the output of dx watch) forwarded to your Splunk instance.
Enabling Forwarding for an Org
Job logs forwarding is an org-level feature. To have this feature enabled for an org, contact DNAnexus Sales.
Setting Up Forwarding
Once this feature is enabled for an org, an org admin with a full-scope DNAnexus token can proceed with setup, following these instructions:
Set up the Splunk HTTP Event Collector (HEC) via the Splunk web interface.
To set up forwarding, use a
dx updatecommand in the following format, substituting the URL of your HEC, your HEC authentication token, and the name of the org:dx update org --job-logs-forwarding-json '{"url":"https://http-inputs-examplecompany.splunkcloud.com/services/collector/event","token":"abc11d22-de34-567f-g8h9-i012jk34lm34"}' org-exampleorgNote that a message is logged to Splunk on successful update of Splunk credentials with the following text:user-xxxx, an admin of org-yyyy is enabling DNAnexus job logs forwarding with these parameters{"url": "https://http-inputs-examplecompany.splunkcloud.com/services/collector/event", "tokenSignature": "<sha256OfSplunkToken>"}Optionally, verify your org's job logs forwarding configuration, using
dx describeas follows:dx describe org-exampleorg --json --verbose |jq .jobLogsForwardingThis will return details of the configuration, in the following format:{"jobLogsForwarding": { "url":"https://http-inputs-examplecompany.splunkcloud.com/services/collector/event", "tokenSignature": "<sha256OfSplunkToken>", "updated": "timestamp-in-milliseconds-from-epoch", "updatedBy": "user-xxxx" }}
Disabling Forwarding
To disable job logs forwarding for an org, use:
dx update org --job-logs-forwarding-json '{}' org-exampleorg
In addition to disabling forwarding for the org, this will record the user who has disabled forwarding, and the timestamp at which this was done, as part of the jobLogsForwarding field for the org. You can see the updated configuration details by entering a command in the following format:
dx describe org-exampleorg --json --verbose |jq .jobLogsForwarding
This will return details of the configuration, which will appear in the following format:
{"jobLogsForwarding": {
"updated": "timestamp-in-milliseconds-from-epoch",
"updatedBy": "user-xxxx" }}
If forwarding has already been disabled for the org, the field jobLogsForwarding will not be updated, with the API returning the code HTTP 200.
How It Works
Once the feature is set up for an org, job logs for all jobs billed to the org will be forwarded to the Splunk instance via the specified Splunk HEC.
The Platform will make reasonable attempts to upload job logs to Splunk in the face of communication errors, while limiting its use of memory and CPU on the worker.
Job logs forwarding takes place outside of the application execution environment. Org admin is responsible for the Spunk instance's ability to accept the volume of job logs generated by jobs in the org.
For cluster jobs, only job logs from the driver node will be sent to Splunk.
Forwarded Job Log Format
Forwarded job log lines will include the folowing Splunk event fields to enable efficient searching and filtering in Splunk:
jobIdThe id of the job generating the log messagetryThe try of the job generating the log messageprojectThe project id in which the job is executingbillToThe billTo of the job generating the log messagelaunchedByThe user id of the job generating the log messagerootExecutionThe job or analysis id of the root execution of which the job is a partexecutableNameThe name of the DNAnexus app or applet being executed by the jobentryPointThe entrypoint in the app or applet being executed by the joblevelThe log level of the jog log line such as EMERG, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFO, DEBUG, STDERR, STDOUT or METRICS
Detailed job metrics are forwarded as a "Detailed job metrics" job log lines that have detailed metrics values attached as event fields named after column headings in dx watch --metrics csv.
Job log lines are truncated to 8015 utf-8-encoded bytes before being forwarded to Splunk.
Job Log Limit
For orgs with this feature enabled, job logs are forwarded to Splunk without being subject to the 4 MiB limit that applies to job logs displayed by dx watch.
Errors
If an error is encountered during job logs forwarding, forwarding will be retried three times. The job's description, accessible by running dx describe for the job, will contain the count of job log lines that could not be forwarded after three (3) retries.
Data Egress Charges
Job logs forwarding incurs data egress charges.
Security Considerations
The customer bears responsibility for managing Splunk users' access to, and protecting sensitive/confidential information contained in, job logs and metadata that have been forwarded to the customer’s Splunk instance as part of this feature. This includes responsibility for compliance with applicable data security and privacy regulations, such as the General Data Protection Regulation (GDPR).
Customer also owns all liability for maintaining the security and privacy of job logs from the point of egress from the DNAnexus Platform to storage within the Customer’s system.
DNAnexus is not responsible for the accuracy of the provided HEC URL.
For more information, contact DNAnexus Sales.
Resource Usage
Customers who have this feature enabled agree to allocate a small amount of worker resources to send job logs to Splunk.
The job logs are forwarded from each worker to the customer's Splunk endpoint via an API request every 30 seconds, or after every 100 job log lines, whichever comes first. Unsuccessful API requests are retried three (3) times. Up to 10MiB of job logs are buffered in the worker's memory for delivery to the Splunk endpoint. If the Splunk endpoint can not keep up with the rate of job log production, some job log lines may be dropped and counted in the job's jobLogsForwardingStatus.linesDropped field. The additional overhead of managing the 10MiB buffer varies from 5 to 15MiB depending on the size of individual job log lines. Job log text is sent to Splunk as JSON strings, along with the metadata associated with each job log line.
Learn More
The video tutorials provide walkthroughs of key org admin tasks:
Last updated
