Discover the best Inference open source repositories on GitHub. Explore trending tools, live star counts, source code, and developer projects updated daily.
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
Use Hugging Face with JavaScript
On-device Speech AI for Apple Silicon
Kyojin: the Yamz inference engine for AMD Strix Halo (ROCm, gfx1151), built on ExLlamaV3. Runs 300B-class MoE models on one 128 GB mini PC.
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.
Runs openpilot's large driving model on Jetson Orin Nano, macOS, iOS, Android, CUDA laptops with only your Comma, and a USB3 cable
TypeDB: Built for systems, not records
NeMo-Speech.cpp is a lightweight C++ inference runtime for Speech models
Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.
llm-d Router: The intelligent entry point for inference requests
An open, programmable decision layer for models and compute.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
Fast ML inference & training for ONNX models in Rust
Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.
Platform for deploying and routing GPU-accelerated inference, streaming, and batch workloads at scale.
A scalable inference server for models optimized with OpenVINO™
Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.
A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability