DeepVariant-on-Spark is a germline short variant calling pipeline that runs Google DeepVariant on Apache Spark at scale.

• DeepVariant is highly accurate. In 2016 DeepVariant won PrecisionFDA Truth Challenge in the best SNP Performance category.
• Apache Spark is a lightning-fast unified analytics engine for large-scale data processing. Apache Spark achieves high performance for both batch and streaming data, using a state-of-the-art DAG scheduler, a query optimizer, and a physical execution engine.
• DeepVariant (v0.7) hasn't supported multiple GPUs. Through DeepVariant-on-Spark, all of GPU resources can be fully utilized across multiple nodes. For example, nVidia DGX-1 has 8 Tesla V100.
• DeepVariant-on-Spark leverages Atgenomix SeqPiper, a wrapper technology using Spark PipeRDD, to parallelize DeepVariant pipeline on Spark and to use Yarn to optimize resource allocation in multi-node environment.

• DeepVariant-on-Spark release notes

• Apache Hadoop 2.8.x
• Apache Spark 2.2.x
• Apache Adam v0.23 (forked and modified by Atgenomix)
• DeepVariant v0.7.0 (forked and modified by Atgenomix)

• DeepVariant-on-Spark quick start on Google Cloud vid DataProc
• DeepVariant-on-Spark WGS case study
• Multiple GPU acceleration
• Customization
• Trobuleshooting

Interested in contributing? See CONTRIBUTING.

DeepVariant-on-Spark is licensed under the terms of the Apache 2.0 License.

DeepVariant-on-Spark happily makes use of many open source packages. We'd like to specifically call out a few key ones:

• DeepVariant - an analysis pipeline that uses a deep neural network to call genetic variants from next-generation DNA sequencing data

• Adam - a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark and Parquet.

We thank all of the developers and contributors to these packages for their work.

• This is not an official Atgenomix product.
• To utilize the official product with full experience, please contact Atgenomix ([email protected]).