txtAI and Python: build and save a semantic search index

This tutorial, uploaded on August 20, 2022, shows how to build a semantic search index with Python and txtAI in a Jupyter notebook. The speaker uses Python 3.7.3 and recommends a 3.7 environment, especially on Windows, because of dependency issues in his setup. No txtAI version is specified; the installation advice describes that period's environment.

The example uses the Sentence Transformers all-MiniLM-L6-v2 model to embed a corpus of about 22,000 accounts of violence and human rights violations in South Africa. The model downloads on first use. The speaker loads descriptions from JSON, prepares records containing an ID, text and an empty tags field, then builds the vector index.

A search for "knife" returns ten results with document IDs and similarity scores. The speaker points to a result about stabbing that lacks the query word, illustrating how semantic search can retrieve related concepts beyond exact keyword matches. This is an example from the demonstrated corpus, rather than a general accuracy benchmark.

The final section saves the index and reloads it in a separate notebook. The original text data also needs to be loaded to print the retrieved descriptions. Streamlit and Google Drive appear as possible topics for a later deployment tutorial; this video covers indexing, querying and persistence.