- Calling /app(let)-xxx/run for Spark apps creates a Spark cluster (+ master VM).
- The master VM (where the app shell code runs) acts as the driver node for Spark.
- Code in the master VM leverages the Spark infrastructure.
- Job mechanisms (monitoring, termination, etc.) are the same for Spark apps as for any other regular app(let)s on the Platform.
- Spark apps use the same platform "dx" communication between the master VM and DNAnexus API servers.
Spark apps can be launched over a distributed Spark cluster.