Inference Server GitHub Repositories (2026) · Open Source Tools

Open-source inference server and production cluster for all the models your agent needs.

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

Runs openpilot's large driving model on Jetson Orin Nano, macOS, iOS, Android, CUDA laptops with only your Comma, and a USB3 cable

The simplest way to serve AI/ML models in production

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

llama.cpp/ik_llama.cpp launcher: loads big MoE models across mismatched multi-GPU rigs by exact VRAM math.

ONNX Runtime Server: The ONNX Runtime Server is a server that provides TCP and HTTP/HTTPS REST APIs for ONNX inference.

One serving daemon for local LLM inference — across runtimes and machines.

Work with LLMs on a local environment using containers