Data Processing GitHub Repositories (2026) · Open Source Tools

Incremental engine for long horizon agents 🌟 Star if you like it!

Apache Wayang is the first cross-platform data processing system.

High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

A light-weight, flexible, and expressive statistical data testing library

Kubernetes-native platform to run massively parallel data/streaming jobs

SQL Runtime without any database

🦾 Mech is a programming language for building data-driven systems like robots, games, and interfaces. Start here!

ESA Earth Observation Toolbox and Java Development Platform

Scalable data pre processing and curation toolkit for LLMs

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

Little utility to decode Metastock files and write them in text format