pgvectorscale

pgvectorscale adds an index for large embedding datasets to PostgreSQL databases that use pgvector. It's for application developers and database administrators who want to keep AI similarity search in their existing database, with more control over search speed and storage use.

Its StreamingDiskANN index draws on Microsoft's DiskANN research for approximate nearest neighbor search. Statistical Binary Quantization compresses vector data to reduce its storage footprint. You can adjust the tradeoff between query speed and accuracy, and it supports cosine distance, L2 distance and inner product using pgvector's query syntax.

Filtering is a particular focus. The index can combine similarity search with label filters, so searches can target categories within a dataset. Other PostgreSQL WHERE conditions work through post-filtering, and you can combine both approaches in one query. Post-filtering processes results as a stream rather than loading the whole result set into memory.

The extension is open source under the PostgreSQL License. In a self-hosted deployment, vector storage and search run in your own PostgreSQL server; Docker containers are available, and source builds support Linux and ARM-based Macs. Timescale Cloud also offers the extension within its hosted database service.

Parallel index building supports compressed datasets without label filters. The search index uses relaxed distance ordering, so returned matches can be slightly out of order by distance.