Top PIPELINE GitHub Repositories & Tools (2026)

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.

An orchestration platform for the development, production, and observation of data assets.

Open-source inference server and production cluster for all the models your agent needs.

Jenkins automation server

Cross-platform, customizable ML solutions for live and streaming media.

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Declarative Continuous Deployment for Kubernetes

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

Flink CDC is a streaming data integration tool

OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg with Table optimization (Called OLake Fusion). ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Supported sources : Postgres, MongoDB, MySQL, Oracle, MSSql, DB2, Kafka, S3.

Shared Single-file memory layer for all your agents, sub mili-second RAG over text, photo and video on Apple Silicon.. No Server. No API. One File. Pure Swift

Tenzir is the data pipeline engine for security teams.

Kubernetes operator to manage installation, updation and uninstallation of tektoncd projects (pipeline, …)

bkit Vibecoding Kit - PDCA methodology + Claude Code mastery for AI-native development

Write your pipelines in C# !

Stroom is a highly scalable data storage, processing and analysis platform.

Streaming AI pipelines in one YAML file. Tokens, audio chunks, and video frames flow between isolated components. Compose 100+ components for models, agents, speech, vision, and live broadcast. Run local models, cloud APIs, or both. Inspired by docker-compose.