Garak

An open-source LLM vulnerability scanner tests local Hugging Face and GGUF models or cloud APIs for prompt injection, data leakage and other failures.

Garak is an LLM vulnerability scanner for developers and security teams assessing models or dialogue systems. It attempts to provoke unwanted responses so teams can identify weaknesses in the systems they test.

Attack probes: Test for prompt injection, jailbreaks, data leakage, hallucinations, misinformation and toxic output using fixed, dynamic and adaptive probes.
Local model testing: Assess generative Hugging Face Transformers models running on your hardware and GGUF models through llama.cpp.
API targets: Test models through OpenAI, AWS Bedrock, Replicate, Cohere, Groq, NVIDIA NIM and LiteLLM, or assess custom REST endpoints that return plain text or JSON.
Failure reports: Review results for each probe and detector, including the rate of unwanted responses, and examine detailed JSONL logs for individual test attempts.

Garak is a free, open-source Python command-line tool licensed under Apache 2.0. Its plugin system supports different model connections, probes and detectors. Local model targets run on your hardware; tests against cloud APIs send prompts to the selected provider. Services such as OpenAI and Replicate require API credentials.

Guardrails, Security and Sandboxing LLM Evaluation and Testing AI red teaming GGUF Hugging Face integration llama.cpp backend