Scikit-LLM with llama-cpp-python: local classification

Scikit-LLM wraps language models in the familiar scikit-learn estimator interface. This walkthrough explains its fit and predict methods, then applies them to zero-shot sentiment classification with a local LLM. The speaker describes support for text classification, named entity tagging, summarization and translation, with local and cloud backends selected through a model string.

The demo uses gemma2-2b-q6, named in the video description, through llama-cpp-python compiled with CUDA. The speaker reports offloading all 27 model layers to an NVIDIA RTX 4090 and using about 2.2 GB of memory. A Gradio browser app loads the model at startup and accepts text plus candidate labels. The example movie review returns the label negative.

The code explanation covers setting GPU layers to minus one for full offload, selecting a local GGUF model, fitting with label names rather than labeled training examples, and calling predict. The speaker presents this setup as an offline workflow without API keys or per-request billing.

Reported performance is around 10 samples per second in steady state. The stated 100% accuracy comes from a quick sanity-check batch, so it does not establish general classification accuracy. The closing comparison weighs GPU memory limits against cloud APIs, which require a network and send data outside the local environment.