Release Notes
Employing a shared image gallery for workers in AWS and Azure.
Turn on autoscaling for the Cloud Manager in the AWS EU-West region only
Correction on measuring the runtime of cluster jobs
Refactoring of data dispensal issues
UI refactoring of Data Manager and Cohort browser to address issues
Rollout of the Ubuntu 18.04 operating system to the internal services of DNAnexus
Apollo: Refactoring reported issues with Apollo and documentation updates
Refactor of worker job limits to allow user to continue launching jobs until their user limit is exhausted in orgs where they have not hit their limit.
Refactoring of association checker function
Exposed data egress monitoring performed by the worker (broken out from overall data egress)
Refactoring of issues of Cloud Manager on Azure spokes
Patching of infrastructure vulnerabilities
Updating the billing information to break out un-archiving charges from the archiving charges
Apollo: refactoring issues
Add currency field to the job_analysis_describe command
Initial fixes for the File Viewer for the Safari viewer (MacOS native browser)
Refactoring of the Cloud Manager log reporting into the SIEM (more information) and log rotation on the Cloud Managers
Apollo: Refactoring reported issues with Apollo, including file status cache issues resolved, fixes to the helper method for exploding columns, etc.
Job Manager and Cloud Manager autoscaling
Detached Job functionality added
Guard rail on number of objects to limit to less than 100,000 objects per command
User Interface fixes to the login redirect URL
Efficiencies for data dispensal application
Apollo: refactoring reported issues
Tools: Original Quality Functionality Equivalent app released - see: https://platform.dnanexus.com/app/oqfe
Refactoring of the Cloud Manager for efficiency and resolution of issues
Refactoring for more efficient communication between the Job Manager and the internal API Server
Make default the view of the archive data usage.
Enable logging of API functions that call other API calls (e.g. API_1 --> API_2 --> API_3)
User Interface Refactoring cleanup
Apollo: Refactoring of reported issues, such as fixes when multiple spark clusters are using the same file, etc.
Refactoring of user describe functionality, lambda updates for performance, fixes to RegionIPRegions to map from the spokes to the user's IP space and improved efficiency of the archive state check functionality
User interface performance fixes - specifically in the data manager
Security vulnerabilities for infrastructure were patched
node.js upgrades in the infrastructure
Apollo: fixes to the scatter plot and dates of the histogram functionality. Fixes to parsing large files (> 2G).
Additional tuning of the Cloud manager for autoscaling
Minor refactoring of "find" API call to return all possible items
Data manager in the user interface will continue to show the size of the objects, but the size will no longer show the size of all related objects
Ubuntu 20.04 is available for workers in all clouds (AWS and Azure) in all regions.
Internal instances of nginx are upgraded to version 1.18
Apollo: Additional refactoring to address issues with the cohort browser, version changes in dxdata, and entity support for boxplot
The planned release was canceled.
Additional tuning of the Cloud manager for autoscaling
Internal refactoring to expand the user describe functionality for spot instances, kill-switch when jobs hit their pricing limitations, and additional reporting from the API servers
Internal nginx servers upgraded to 1.18
Apollo: additional refactoring to the cohort browser
Update of cloud Manager for autoscaling
User Interface refactoring of known issues
Apollo: Refactoring of locus controls, genomic range controls and decimals in the histogram
Refactor cloudmanager for VPC changes
Refactoring of the operation of cluster computing in workflows
Apollo: fixes to handling netstat error. Refactorig of the cluster packages and use of the appropriate signatures in Apollo components.
Apollo: go live of Apollo in the London cloud for select customers.
Refactoring links between the Job Manager and API server
Added DNA_API_PROVDER_DESCRIBE functionality to the Platform
Addition of AWS-London cloud to DNAnexus environment for specific customers
Vulnerability patching of the DNAnexus cloud infrastructure
Apollo: Refactoring of V3 of data/time and max/min range of the histogram. Added boxplot explod functionality for hierarchical and multicolumn selections
Refactoring of the autoscaling within the Platform
User Interface refactoring (restoration of breakcrumbs in the data manager and highlighting)
Apollo: upgrade of the VisZerver to the V3 distribution
Cloud Manager refactoring
Updates to Grub2 and other security fixes
Upgrades to Docker containers/pilot
Apollo: backend changes to Histogram, ScatterPlot and descriptor functionality
Updates to internal servers - API Servers to Ubuntu 20.04
Refactoring of database cluster in the infrastructure
Refactoring of the project describe functionality to address issues
Okta-SCIM updates (SSO for customers)
Apollo: internal code to perform additional data payload verification tests, cohort code refactoring and updating dx-data in Docker.
User Interface Refactoring, including adding users to org by email, fixes to password complexity, UI footer style checks and a fix for a race condition occasionally seen when selecting folders.
Cloud Manager refactoring
Proxy Lock refactoring for the China region only.
OktaSCIM changes
Updates to the external SSO - Okta SCIM
Fix for the SQS SendMessage batch message length to allow for larger messages
User Association Page
Apollo testing framework update
Refactoring known user interface issues, fixes to data manager
Routine updates of NVIDIA drivers for the AMI instances that use GPUs (includes security update)
Fixed occasional bugs in upload proxy; fixes to detached job issues
Apollo: Refactoring of code to support V3
Enhancements for internal robustness of the https-worker client
User interface refactoring, including changing the load spinner to reflect current brand colors
Apollo: align ownership of internal files for smoother processing, updating panda and internal query routes. Updates on v3 of the updater and data loader.
Fix for jobeventsconsumer code for old objects
Completed fix for JobManager API calls
Update to https-workder to allow custom names
Apollo: dxdata refactoring and added compatibility check when uploading data
Refactoring of the internal platform message queue to decrease the number of calls from the redis server.
Increase platform robustness by refactoring the jobEventQueue and Standard Queue
User Interface refacotring in the data manager
Apollo: Refactoring cluster log collection and arguments of the vizserver API. Fixes in the backend components to support the allele frequency functionality.
UI refactoring ("pannexin"), including deprecation of unused code, fix for missing file size in file preview for image/pdf, refactoring of the data manager and fix of the range filterin in GWAS Manhattan dataset.
Added government required links regarding contacts and abuse to the China cloud of DNAnexus.
Refactoring of log rotation within the Platform
Apollo: Changes of the vizserver to support the new software versi0ons in the backend, API refactoring and collection of logs in the Spark cluster.
Minor bugfixes in the Platform, such as resolving symlinks, etc.
User Interface switch between the legacy UI "Membrane" and newer UI "Pannexin"
Disable public user from executing a user-describe functionality (security enhancement)
FedRAMP email channel through Salesforce is activated
Common password rejection
Apollo: refactoring of API design, update error messages, date & time fixes, and upgrade to the HAR files
Refactoring Cohort Comparison Browser
Move the legacy authentication server to Python 3
Security fix for stylesheets
Updates to the Okta SCIM SSO server
Refactoring of the nginx web server on the worker AMIs to resolve existing issues
Apollo: refactoring geno cohort filter
Cookie notice will appear in compliance with EU requirements
Minor fixes to node.js installers and packaging (internal)
Refactoring of existing Cloudmanager to increase the monitoring scope
Apollo: Added DNA_GUI_SURVIVAL_PLOT functionality in the cohort browser. Added support on Azure for the Thrift server integration from VizServer
Refactoring of Cohort browser and Companion browser, including max/min labels on charts, color changes in some icons
Infrastructure refactoring to node.js installers and packaging.
Internal refactoring of the Azure Cloud Manager to handle docker containers
UI refactoring for project list, set bill-to components, and cohort browser/comparison
Apollo: added DNA_GUI_COHORTCOMPARE_LOAD functionality
Upgrade internal platform to Mongo 3.6 (in compatibility mode)
UI changes to Manhattan plot zooming and refactoring of Cohort Browser/comparisons
Apollo: Added TIP_APPS_PLATO_GWAS functionality
New Lambda function for internal autoscaling
Storage accounting updater daemon added (internal only)
Upgrade the internal version of Python from 2.7 to 3.x
Refactoring Pannexin User Interface (e.g. fixing zeros that showed up in preview, but held values)
Update Okta-SCIM infrastructure
Refactoring cohort browser
Upgrades and refactoring of the Cloudmanager on the Platform
Refactoring to existing cohort browser (e.g. list, nodeSearch and cohort comparison paging)
Apollo: minor refactoring to address issues in the backend.
Refactoring to existing cohort browser
Security patches to the user interface
Added the following functionality: DNA_GUI_DM_ELEMENT_CHECK, DNA_GUI_DM_NAVIGATION, DNA_GUI_SIDEBAR_OPEN, DNA_GUI_DM_INFO_TOGGLE, DNA_GUI_CUSTOMIZE_COLUMNS, DNA_GUI_FILTER_FIELDS, DNA_GUI_DM_CLEAR_FILTERS, DNA_GUI_DM_LOAD, DNA_GUI_DM_CREATE_NEW_FOLDER, DNA_GUI_DM_CREATE_NEW_WORKFLOW, DNA_GUI_DM_UPLOAD_FILE, DNA_GUI_DM_COPY_FROM_PROJECT, DNA_GUI_DM_START_ANALYSIS, DNA_GUI_DM_DEFAULT_ACTIONS
Apollo: Added the following functionality: DNA_GUI_REMOVE_ALL_TILES
Self-service archiving on Azure
Python 3 rollout in Platform infrastructure
Apollo: refactoring of existing code
Platform - refactoring of known issues
Apollo: additional data loader functionality
Minor refactoring on the Platform and infrastructure for Azure and AWS clouds
Apollo: minor refactoring of outstanding bugs in Vizserver
Upgrade internally to Ubuntu 16.04 (from Ubuntu 14.04)
Refactoring of Apollo - minor bugfixes
Job Manager changes to allow Job Manager to talk to internal API servers via tokens
Cluster job failure resolution - if there are multiple failures, will quit
Refactoring of cohort browser at the UI level
Apollo: Update to Python 3.7 in the backend
Refactoring s3deleter code
Refactoring internal billing code
Cloud Manager changes to support GPU instance types
Pricing code changes for multi-region AWS archiving
Apollo: Refactoring histogram functionality
Apollo: Implementation across Azure (not just AWS)
Refactoring of archive modal iconf, data catalog page presentation fixes, UI efficiency improvements, add support for number format for 'e(+/-)', fix for the medusa update, data limits message clarified, documented tile requirements, fix cohort browser UI to allow dashboardConfig without any geno data components, fix filter pills overflow, remove usage of limits for charts from fieldsJSON and parallelize jenkins tasks.
Selective release of new functionality to support relational databases
Apollo: Minor refactoring of sorting file parts before joining, link datasort from selected cohorts, set reverse DNS lookup to false, store genoBins in cohort details and update specific multi-part upload configs on Azure
Replace old DNAnexus logo on error pages
Remove old archival code
Apollo: refactoring binBy histogram code, license check code, and tipsync feature
Additional instances types added for DB Cluster
File archiving in China
UI changes
File archiving refactoring
Apollo: filter options
Apollo: Looker sunset
Refactoring User Interface to include SSL headers
Pages listing org-admins moved to the Pannexin user interface
Enable GPU instances in China cloud
Refactoring of the DB Cluster code
Refactoring of the file archiving code
Apollo: Minor refactoring of code and documentation
Extended refactoring of file archiving in areas such as timeouts, increased efficiencies, etc.
Add license checks to ensure files can be downloaded
Refactoring of the cohort browser user interface.
Horizontally scaling CloudManager to handle more concurrent workers by sharding within each instance type.
Release of autoscaling of the authorization server to allow for high availability as the usage increases
Apollo: Refactoring of select queries, trimming of caches, etc.
Refactoring of billing code with respect to file archiving (internal functionality)
Additional code to ensure the filenames of files to be archived have not already been archived
Apollo: refactoring of Basic 2 variable charts
Refactoring self-service archiving, including the user interface and checks to make sure the files to be archived have not been deleted since the archiving process started
Refactoring the CloudManager on AWS to handle more concurrent workers (via load balancers)
User Interface refactoring in pannexin for the cohort browser and customer-specific browsers
Apollo: minor refactoring for database support on Azure.
Refactoring of the backend systems supporting Titan, including bugfixes for improvements, customer-specific user interface encements
Apollo: Additional code buildout for local processing on Azure clouds, removal of code to acess Looker (depracated visualization module).
Refactoring of the user interface for improvements, customers-specific user interfaces.
Apollo: code buildout for local processing on Azure clouds, improvements in the cohort browser
User Interface changes, including the overall look and feel, ain project folders and data selector.
Refactoring in Titan, including refactoring Mongo queries, additional internal error messages, etc.
Refactoring in Apollo around documentation and Mongo queries
Add authentication token and labels at the end of the human readable audit trail
Change and add new instance clases (c.3.xlarge becomes c5.xlarge; new instance c5.9xlarge)
Horizontal scaling of CloudManager to handle more concurrent workers (AWS only)
Apollo: Minor refactoring of code supporting the viz server, API servers for list of folders and dxdownload support
Minor changes to the Platform U(ser) I(nterface) to support the use of the visualization server in Apollo
Backend ability to push object to the archive (AWS only)
Apollo: Minor changes to the vizserver (visualization server)
Refactoring dbcluster - fix where dbcluster gets stuck in "creating fix" step
HITRUST certification received
Feature switches - internally focused on billing
Audit Log filenames - only for customers with "human readable" audit logs, files will have a prefix of YYMMDD to allow natural collation
Apollo - revisions to cohort browser to enable multi-cohort charts
Apollo - integrate additional variant annotation sources
Refactoring dbcluster code
Refactoring of job manager user interface - should be faster
Cleanup of the backend (not accessible to users) for Chart Builder, including support for importing CSV files
Apollo: Cleanup and removal of mapping of Spark (full transfer to Spark 2.2)
User Interface - removing featured projects when there are no projects
Refactoring internal docker code
Refactoring internal dbcluster code
Apollo: batching filesystem deletes
Refactoring of memory lead issues related to file closers
Refactoring of user interface for samples management
Moving wiki links hardcoded into the UI to the documentation.dnanexus.com site
Apollo: SQS queuing for vizsync service
Apollo: users can add multi-variate charts to the cohort browser
Apollo: Extending API Service module to obtain the SQL query from the Visualization Server
Apollo: Refactoring of occasional 404 error from the Thrift Server
Add DBCusterStart (relational database functionality) in CloudWatch
Refactoring to correct asserion error when using findDataObjects when using SortBy filed.
Expanded damemon to abandoned deleted files from a maximum of 1024 files to unlimited, after initial warning
Refactoring race conditions in jobs finding files in workstation.
Refactoring code for the community selector when the user has no communities
Migrating Jenkins nucleus jobs from Ubuntu 14.04 to 16.04
Downloading is stopped if the customer's account is out of money
Apollo Refactoring: Disable functions that are no longer used (e.g. reload function and getFunctions)
Apollo Refactoring: Fix dataset sync for visualization between the API server and visualization serve
Apollo Refactoring: Log the start/end time between API calls
Apollo Refactoring: Fix the project feature access check
Move lambda functions from node 6.10 (end of life) to 10.x
Remove project file closer locking as this is no longer necessary
Move lambda functions from node 6.10 (end of life) to 10.x
Add archival billing events to stream to the billing apps
Modify community login and registration pages to current branding as well as hidRegistration option
Fix the page loading of Internet Explorer 11 (11.1087.16299.0) as well as an older version of IE (11.0.9600.1784.eIS)
Apollo - Code to support sharding of relational databases. Default will be "off"
Apollo - SQL fixes to Looker
Apollo - Removing default of using vizservr as the default visualization service. User will need to specify visualization server
Refactoring header authorization in the user interface
Refactoring the code supporting community links on the Platform user interface
Update dx-toolkit to 0.281.0
Fix the internal tagging of the docker images on the Platform
Minor revisions to the navigation bar in the header
Internal replacement of the MongoDB sotrage system from MMAP technology to WiredTiger technology
Refactoring of unarchive functionality
Slight styles changes to the navigation bar (no changes to the functionaly). User interface will move to v4 of WebPack.
Apollo - bring your own database (BYOD) to now include JDBC drivers for Postgres, msSQL, MariaDB, MySQL, Amazon Redshift and Oracle
Apollo - security update for Looker
Slight changes to the Platform toolbar
Okta SSO account migration
Refactoring costing model for the arcidAT command
Refactoring of archive
Create docker service skeleton
Apollo move throttle to the server
Refactoring of workiner execution for slow jobs
Refactoring of nucleus issues (fix of eventconsumer logging defect)
Cohort Browser is move to the standard Platform interface
Apollo refactoring to return runtime file-ID from dashboards for saving cohorts
Apollo Looker security update from v5.16.13 to v6.8.27 (latest)
Refactoring LXC address documented change from LXC 2.0.8 to LXC 2.0.11
Added larger instance type c5d.9xlarge
Fixed NVIDIA driver load for p3.2xlarge instances
Clean up execserver code to address log truncation
Enable /dev /tmpfs setup and clean up LXC
Add archivedAt timestamp during the CloudArchival process
Apollo - pooling http client connectiopns for APIserver calls
Apollo - refactor connection pool configurations for APIserver client on thrift and metastore
Pannexin Tool Library Changes: improvements to the Tools Library, new user experience for the Tool Library whereby the information in /apps will be contained /panx/tools. /apps will go away; Tools Library now includes Global Workflows; Apps and Global Workflows launched through Tools Library will load in the new Tool Runner by default.
Enable trusted org-admins to lock/unlock, reset MFA for their org users.
Web user interface can now run global workflows
Additional fixes to DBCluster
Provide a public IP address per worker on the Azure environment
Port of Jobmanager to a docker build base, which is based on Alpine
Upgrade Docker to 18.09.5. Simplify docker install
Separate the license from Jupyter Lab from Apollo so they can be licensed separately
Refactoring of code for the cloud archival containers
Integrate JupyterLab into the menu functionality of the web interface for the Platform
Increase Node.js memory limit for the job log archiver service.
Implement a container function to manage the archive process.
Refactoring of Apollo code to address issues
Refactoring of findProject API call
Support loading and interactive querying of full whole exome samples in Apollo
Unique ID of the Thrift Server used written to the audit log (Apollo)
Support for single node spark clustger (instead of requiring a minumum of 2 nodes; Apollo)
Updating the DNAnexus logos to new branding, primarily on CSS sheets of web pages
Detect and report worker filesystem errors in the worker log file
Refactoring of code to make sure archiving does not include symlinks
fix for datbase sort memory limit with findProject memory limitation
File leve archive for API and CLI support to archive and unarchive files, folders and projects
Fix to completely delete the files in scope on Auzre
Fixes for the internal dbCluster database files
Pooling fo services on Apollo so that small jobs doe not get caught behind large jobs.
Tuning of Prometheus sensing software
Apollo refactoring of reconnection, fix for ORC file formats and fix to make sure the error messages are correct.
Platform to restrict the running ot HTTPS apps
Platform to upgrade NVIDIA driver on relevant workers
Drone moved to the same VPC as Artifactory (no functionality change)
Implement new billing eents database
fixes to bill for file archiving
Refactoring of the MongoDB timeout parameters
Removing obsolete job group logging code from Apollo
Refactoring of filter in the getUSER api call
Refactoring the m3.medium ephemeral disk space issue
fixing learna for archiving common packages
Updating the unarchived API logic
adding autoscape hosts to perform platform_pilot_status
Adding internal metrics and alerts to the internal audit trails from the Platform.
Refactoring the error handling in the MongoDB for metadata storage
Phase out usage of describeSpotInstanceRequests as this is no longer being used
Refactor internal account for cluster jobs
Refactor the file status caching logic for Apollo
Refactor internal authentication server
Refactoring Azure-specific code for the Platform to make more efficient.
Refactoring the filecloser functionality to address bugs.
Additional monitoring code with Prometheus added to file uploader.
Continued migration of existing SSO users to Okta SSO
Concurrent Org Worker Limits:All orgs that do not have custom concurent worker limits will default to a maximum of 500 concurrent workers.
Refactoring of API servers for Okta tokens (impacts only customers moving to Okta SSO)
Refactoring of Nucleus for compatibility with MongoDB 3.4
Refactoring to fix bugs in dx_download
Updates to the SCIM server for specific customers to integrate with the DNAnexus Platform Okta authentication server (no impact for all other customers)
Deprecated command line interface (CLI) commands "dx sh" and "dx exit"
Internal refactoring of code, including adding internal monitoring and alerting functionality
Added i18n/I10n support to allow the user interface to recognize the user's language selection
Refactoring external API servers to support Ubuntu 16.04 and configure for autoscaling
Ability of the user to archive without requiring support intervention (AWS-East only)
Augmentation of the MongoDB server with AWS Aurora server (AWS only) for select customers
Support for HTTPS access for JupyterLab
Updated the download proxy service to handle large volume of big files
Movement of SSO functionality from the internal Auth Server to Okta for an existing customer
Custom expiration dates on tokens
User's profile now allows email preferences
Turned on the load balancer on the internal API server.
Upgraded the nginx webserver to V15
Refactoring of the job manager to a high availability status
Allowing users to see the public symlink URL
Upgrade to REACT to V16 of the Javascript library
Removed the social media links on the landing page of the Platform
Alerts to org admins when the audit trail is turned off (a configuration setting)
Deployment of the Apollo functionality of the Translational Informatics Suite to appropriately licensed users
Refactoring code for data egress
Refactoring of the beta-Apollo code (Translational Informatics Suite)
Refactoring of data egress functionality
