UQLM

UQLM is a Python library for developers and researchers assessing the reliability of LLM answers. Its confidence scores estimate the likelihood of errors or hallucinations; they don't verify that an answer is factually correct.

• Response consistency: Compare multiple answers to the same prompt without needing model internals, and select the answer with the lowest estimated uncertainty.
• Token probability scoring: Assess a single generated answer without extra LLM calls when the model exposes token probabilities.
• Judges and ensembles: Evaluate responses with a chosen LLM or panel using customizable prompts, or combine scorers with fixed weights or weights learned through supervised training.
• Claim-level review: Score individual claims or sentences in long answers and rebuild a response after removing claims below a confidence threshold.
• Generated code assessment: Estimate functional correctness without executing the code, using token probabilities, functional-equivalence comparisons and model self-evaluation.

UQLM is open source under Apache 2.0 and works with LangChain chat models, including Ollama for local LLMs and cloud backends such as OpenAI, Vertex AI and Azure OpenAI. Where model requests go depends on the backend you choose. Consistency and long-text scoring require multiple generations, adding latency and model calls; token probability scoring requires a backend that exposes those probabilities.