Pathway

Pathway's Live Data Framework is a Python framework for developers building RAG applications and analytics over changing data. It runs locally or on your own servers, including through Docker. Its incremental engine updates results as data arrives, so applications can work with changing documents and events.

The same pipeline code handles batch data and continuous streams. A Rust engine based on Differential Dataflow executes the computation, while developers use Python functions and ML libraries for transformations. Stateful operations include joins, time windows and sorting. Late or out-of-order data can trigger corrections, and persistence lets a pipeline recover saved computation state after a restart.

For AI applications, Pathway includes document parsers, embedding and text-splitting utilities, and an in-memory vector index that updates with incoming data. It integrates with LangChain and LlamaIndex. Local model examples use Ollama; other examples use cloud models. Requests to external model or data services leave your deployment.

Connectors include Kafka, PostgreSQL, Google Drive and SharePoint. Airbyte extends the available sources, and a Python connector interface supports custom sources. Monitoring exposes connector activity, latency and logs.

The framework is source-available under BSL 1.1. Its additional production grant permits one machine per installation within the runner's resource limits and excludes offering a hosted stream-processing service. Distributed production deployment and other uses outside that grant require an appropriate license.