Inference GitHub Repositories (2026) · Open Source Tools

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.

A high-performance inference engine for AI models

Port of OpenAI's Whisper model in C/C++

A high-throughput and memory-efficient inference and serving engine for LLMs

Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.

Cross-platform, customizable ML solutions for live and streaming media.

Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

A unified inference and post-training framework for accelerated video generation.

State of the art inference performance with modern accelerators on Kubernetes

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

A framework for efficient model inference with omni-modality models

The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

Open-source inference server and production cluster for all the models your agent needs.

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Manages Unified Access to Generative AI Services built on Envoy Gateway

SGLang is a high-performance serving framework for large language models and multimodal models.

Infrastructure for continually self‑improving agents

An open-source inference engine for AI functions in SQL