Spark GitHub Repositories (2026) · Open Source Tools

Python SQL Parser and Transpiler

Apache Spark - A unified analytics engine for large-scale data processing

Apache Auron is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing.

LLM speculative inference server for heterogeneous hardware & consumer GPUs

Apache Wayang is the first cross-platform data processing system.

Skardi is a self-improving context framework for ai agents

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

Apache Spark Kubernetes Operator

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

LakeSoul is an end-to-end, realtime cloud-native Lakehouse framework for fast data ingestion, concurrent updates, incremental analytics, multimodal data processing and vector search — powering next-generation BI and AI workloads.

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

YTsaurus is a scalable and fault-tolerant open-source big data platform.

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

Scalable master data management, identity resolution, entity resolution, and deduplication using ML

Apache Amoro(incubating) is a Lakehouse management system built on open data lake formats.

Cross-platform real-time collaboration client optimized for business and organizations.

Firely and Incendi's open source FHIR server