TrustLLM

TrustLLM is a research toolkit for teams comparing language models beyond answer quality. It evaluates local LLMs and models accessed through APIs for behaviors such as hallucination, harmful responses, bias, and information leakage.

• Trustworthiness tests: Assess truthfulness, safety, fairness, robustness, privacy, and ethics through task-specific datasets and metrics.
• Local models and APIs: Evaluate Hugging Face Transformers causal language models, existing checkpoints, or text-only Chat Completions endpoints, including local OpenAI-compatible servers.
• Safety and reliability checks: Test jailbreak resistance, harmful misuse, unnecessary refusals, factual errors, and responses to adversarial or unfamiliar inputs.
• Scores and research records: Keep evaluation data, settings, and results together, and review scores in HTML or JSON; scoring rejects incomplete response sets.
• Agent orchestration: Use the Python interface or CLI with machine-readable results to include evaluations in AI agent workflows.

The Python toolkit is open source under the MIT license. Local model loading supports CPUs, CUDA GPUs, and Apple's MPS backend, subject to model compatibility and available memory. Hugging Face weights download on first use unless you supply an existing checkpoint.

Evaluation isn't guaranteed to stay fully local: some scoring tasks use downloaded classifiers or embeddings, while others send requests to an API judge that requires provider access and may incur charges. The benchmark covers six dimensions; transparency and accountability belong to its broader framework but aren't benchmarked.