Top BIG-DATA GitHub Repositories & Tools (2026)

ClickHouse® is a real-time analytics database management system

Apache Spark - A unified analytics engine for large-scale data processing

Apache DataFusion SQL Query Engine

Open-source IoT Platform - Device management, data collection, processing and visualization.

Apache Fluss is a streaming storage built for real-time analytics.

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Apache Beam is a unified programming model for Batch and Streaming data processing.

Apache IoTDB

Apache Hive

CrateDB is a distributed and scalable SQL database for storing and analyzing massive amounts of data in near real-time, even with complex queries. It is PostgreSQL-compatible, and based on Lucene.

Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF)

Apache Auron(Incubating) is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing.

Distributed Big Data Orchestration Service

Use CH-UI to work with your data from Click House self-hosted with a user-friendly interface. CH-UI is a modern and feature-rich user interface for ClickHouse databases. It offers an intuitive platform for querying ClickHouse databases, executing queries, and visualizing metrics about your instance.

Stroom is a highly scalable data storage, processing and analysis platform.

Knowage is the professional open source suite for modern business analytics over traditional sources and big data systems.

Apache Paimon Rust The rust implementation of Apache Paimon.

Collaborative Datacenter Simulation and Exploration for Everybody

Clojask is a Clojure data processing framework with parallel computing on larger-than-memory datasets