Lance

Lance is an open source data format for teams building search engines, feature stores and machine learning training pipelines on object storage. It keeps images, video, audio, text and embeddings alongside tabular data, so those workloads can share a dataset. The project uses the Apache 2.0 license.

Its search capabilities combine vector similarity, BM25 full-text search and SQL filters. Indexes are part of the format, rather than a separate search layer. Fast access to individual records supports training samples, model serving and interactive applications, while scans support analytics across larger datasets. Large media objects can load only when needed.

Feature engineering doesn't require rewriting an entire table each time you add populated columns. Lance can add embeddings or other derived values by writing new files, and it supports ACID transactions and time travel for managing changes to datasets. Stable storage versions have a compatibility guarantee: future releases continue to read data written in those versions.

Lance includes file and table formats plus a catalog specification. Its core is written in Rust, with Python and Java bindings. It works with Pandas, Polars, PyArrow, Ray and PyTorch for data processing and training; SQL and distributed processing integrations include DuckDB, Apache Spark, Trino and Apache Flink. Catalog integrations include Apache Polaris, Unity Catalog, Apache Gravitino and Hive Metastore.

The pylance Python package reads and writes datasets locally; the project also offers Rust bindings. Lance is the storage format and developer SDK, rather than a standalone database server.