Chonkie
An open-source RAG ingestion library for Python and JavaScript, with local processing, a self-hosted REST API, and an MIT license.
Chonkie is an MIT-licensed, open-source library for developers preparing documents for retrieval-augmented generation (RAG). It combines text cleaning, chunking, embeddings and vector database ingestion, so you don't have to assemble each stage from separate libraries. It runs locally or in the cloud, with Python and JavaScript support and a self-hosted REST API that can run in Docker.
Its chunking methods include fixed-size token splits, sentence-based splits and semantic chunking. A byte-based chunker uses SIMD acceleration for fast text splitting. The library keeps its dependency footprint small by making specialized components optional, and it supports multilingual text.
Document processing covers local files and directories, structured Markdown, and CSV or Excel tables. After splitting, Chonkie can merge overlapping chunks based on similarity and attach embeddings through a chosen provider. Reusable pipelines connect these stages, while vector database integrations handle ingestion. You can also export chunks as JSON or HuggingFace datasets.
Local processing and the self-hosted API run on your own hardware. Some integrations use external services: Mistral OCR sends images or PDFs to the Mistral OCR API for text extraction. Chonkie also provides skills for coding assistants including Claude Code, Cursor and Copilot, covering its APIs and RAG pipeline patterns.
