Apollo Apps

Spark Applications
A license is required to use Spark on the DNAnexus Platform. Contact DNAnexus Sales for more information.
The Spark application is an extension of the current app(let) framework. Currently, app(let)s have a specification for their VM (instance type, OS, packages). This has been extended to allow for an additional optional cluster specification with type=dxspark.
Calling /app(let)-xxx/run for Spark apps creates a Spark cluster (+ master VM).
The master VM (where the app shell code runs) acts as the driver node for Spark.
Code in the master VM leverages the Spark infrastructure.
Job mechanisms (monitoring, termination, etc.) are the same for Spark apps as for any other regular app(let)s on the Platform.
Spark apps use the same platform "dx" communication between the master VM and DNAnexus API servers.
There's a new log collection mechanism to collect logs from all nodes.
You can use the Spark UI to monitor running job using ssh tunneling.
Spark apps can be launched over a distributed Spark cluster. 
Using Spark
Connect to Thrift
Last modified 8mo ago