Data Engineering GitHub Repositories (2026) · Open Source Tools

Business intelligence as code: build fast, interactive data visualizations in SQL and markdown

Text-to-SQL access control: schema selection that gives the LLM only the tables the caller is allowed to read. Row-level-security-aware table pruning for LangChain, MCP and any SQL agent. Postgres, Oracle, MySQL, SQL Server. 75% fewer prompt tokens.

Workflow Engine for Kubernetes

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

An orchestration platform for the development, production, and observation of data assets.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Graph-Native Infrastructure for Context and Accountable AI Systems

Job board aggregator indexing 1,000,000+ active positions from 20,000+ companies across Greenhouse, Lever, Ashby, Workday, and other major ATS platforms. Multithreaded Python ETL pipeline, daily automated refreshes via GitHub Actions, filterable client-side search.

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

Apache Superset is a Data Visualization and Data Exploration Platform

Incremental engine for long horizon agents 🌟 Star if you like it!

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

Installer and quickstart for DataKitchen's open-source DataOps TestGen and DataOps Observability. Installs TestGen with Docker Compose or pip, Observability with Docker Compose.

Maestro: Netflix’s Workflow Orchestrator

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows