Nlp GitHub Repositories (2026) · Open Source Tools

A comprehensive Rust translation of the code from Sebastian Raschka's Build an LLM from Scratch book.

🎤 vibrato: Viterbi-based accelerated tokenizer

Additional packages (components, document stores and the likes) to extend the capabilities of Haystack

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

NLTK Source

ModelScope: bring the notion of Model-as-a-Service to life.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Superfast AI decision making and intelligent processing of multi-modal data.

Module for automatic summarization of text documents and HTML pages.

Extract structured data and insights from documents using LLMs and decision models

Collection of open-source libraries and tools for Robotic Process Automation (RPA), designed to be used with both Robot Framework and Python

A Framework of Small-scale Large Multimodal Models

Pretrained model hub for Keras 3.

The Jieba Chinese Word Segmentation Implemented in Rust

Tock, the open source conversational AI toolkit.

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).

potato: the portable annotation tool

Open language modeling toolkit based on PyTorch

OculiX — Visual Automation IDE. If you can see it, you can automate it.