Apollo Apps

Spark Applications

A license is required to access Spark functionality on the DNAnexus Platform. Contact DNAnexus Sales for more information.

The Spark application is an extension of the current app(let) framework. App(let)s have a specification for their VM (instance type, OS, packages). This has been extended to allow for an additional optional cluster specification with type=dxspark.

• Calling /app-xxxx/run or /applet-xxxx/run for Spark apps creates a Spark cluster (+ master VM).

• The master VM (where the app shell code runs) acts as the driver node for Spark.

• Code in the master VM uses the Spark infrastructure.

• Job mechanisms (monitoring, termination, and management) are the same for Spark apps as for any other regular app(let)s on the Platform.

• Spark apps use the same platform dx communication between the master VM and DNAnexus API servers.

• You can use the log collection mechanism to collect logs from all nodes.

• You can use the Spark UI to monitor a running job using ssh tunneling.

Spark apps can be launched over a distributed Spark cluster.