LLM Serving GitHub Repositories (2026) · Open Source Tools

A high-throughput and memory-efficient inference and serving engine for LLMs

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.

The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Tensor Fusion is a state-of-the-art GPU virtualization and pooling solution designed to optimize GPU cluster utilization to its fullest potential.

Community maintained hardware plugin for vLLM on Huawei Ascend

Can AI Agents Build Bespoke Systems?

One serving daemon for local LLM inference — across runtimes and machines.