Best NLP & Language Models GitHub Repositories

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

12 Weeks, 24 Lessons, AI for All!

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

NLTK Source

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.

ModelScope: bring the notion of Model-as-a-Service to life.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Local-first healthcare AI: clinical NER and HIPAA PII de-identification on hardware you control. 2,200+ medical models, 35 model-backed PII languages, and Python, MLX, Android and browser runtimes. Apache-2.0 SDK.

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Superfast AI decision making and intelligent processing of multi-modal data.

Module for automatic summarization of text documents and HTML pages.

Open-source inference server and production cluster for all the models your agent needs.

textlint is the pluggable linter for natural language text.

Extract structured data and insights from documents using LLMs and decision models

Collection of open-source libraries and tools for Robotic Process Automation (RPA), designed to be used with both Robot Framework and Python

Neural Network Compression Framework for enhanced OpenVINO™ inference