Deduplication GitHub Repositories (2026) · Open Source Tools

Ephemeral Workspaces, durable shared history. LayerFS gives every agent a complete, writable filesystem Workspace over shared committed history, without copying the base. Agents work in parallel, commit what changed as deduplicated checkpoints, then branch, rewind or merge. Try it with LayerFS × DSH.

Windows disk usage analyzer and cleanup tool with interactive treemaps, large-file discovery, duplicate detection, and file-system monitoring.

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

Prometheus Alertmanager

plakar is a backup solution powered by Kloset and ptar

Replace complex Borg Backup terminal commands with a beautiful web UI. Create, schedule, and restore backups with just a few clicks.

Context intelligence layer for LLM agents: persistent memory with write-time dedup, sensitivity tagging, conflict detection, and hierarchical decay. ~12ms. No LLM calls. MIT.

A Spicetify extension that adds powerful sorting, filtering, UI features, and more to the Spotify desktop client.

Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

Fast, secure, efficient backup program

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

Extremely fast tool to remove duplicates and other lint from your filesystem

Scalable master data management, identity resolution, entity resolution, and deduplication using ML

Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.

Scalable data pre processing and curation toolkit for LLMs