ModernBERT

ModernBERT is a family of text encoder models for developers building search and classification systems on their own hardware. The base checkpoints predict masked words; retrieval, classification and question answering require task-specific fine-tuning.

• Long document inputs: Models accept up to 8,192 tokens, so applications can process longer documents and code passages than the 512-token limit common in earlier BERT encoders.
• Code retrieval: Training includes substantial source code data, making the models relevant to search tasks that combine code and natural language.
• Retrieval training: Examples support dense retrieval with Sentence Transformers and ColBERT models through PyLate, giving developers two approaches to adapting the encoders for search.
• Model research: The research code supports pre-training, GLUE tests and retrieval evaluation, while FlexBERT lets researchers build encoder variants.

Base and large checkpoints work with Hugging Face Transformers. Compatible GPUs can use Flash Attention 2 for faster processing; performance evaluations include an NVIDIA RTX 4090. The research code builds on MosaicBERT and uses the Apache 2.0 license.