Gensim

Gensim is a free, open-source Python library for developers and researchers working with large text collections on their own machines or servers. It trains models that capture topics and semantic relationships, then uses those representations to find related documents. It runs on Linux, Windows and macOS under the GNU Lesser General Public License v2.1 (LGPL-2.1).

Its main distinction is how it handles large datasets: Gensim streams text through its algorithms, so the full collection doesn't have to fit in RAM. That matters for projects analyzing publication archives, legal documents or other collections that exceed available memory. Optimized C and Fortran routines and multicore processing handle the computational work behind its Python interfaces.

Gensim includes LDA and Latent Semantic Analysis for topic modeling, plus word2vec for learning word embeddings. Document indexing and similarity retrieval let applications compare a query with a collection of texts. These capabilities suit semantic search, recommendation systems and analysis of recurring themes in documents.

Developers can connect their own text streams and extend the library with other vector-space algorithms. LSA and LDA can also run across a cluster of computers. The Gensim-data project supplies pretrained models and corpora, including models for legal and health domains, and Gensim can read training data from remote storage such as S3.