Model Serving GitHub Repositories (2026) · Open Source Tools

A high-throughput and memory-efficient inference and serving engine for LLMs

OpenLake is a high performance storage engine for efficient LLM inference and GPU Training

A framework for efficient model inference with omni-modality models

Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s — in pure Python with async/await, also other languages (rust, go and ts)

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

The simplest way to serve AI/ML models in production

A scalable inference server for models optimized with OpenVINO™

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Open-Source Personal Cloud OS for Always-On Agents

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

High Performance Serving Runtime For Interactive Super Intelligence System

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Community maintained hardware plugin for vLLM on Huawei Ascend

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

Fast, concurrent inference for YuE2. Same model and recipe: 1.68× faster per song, 3.31× more songs per GPU.

An open-source virtual AI Infra team that discovers, benchmarks, and safely upgrades local LLM inference—with independent quality gates, a stable OpenAI-compatible API, and automatic rollback.