NVIDIA NeMo Guardrails on Red Hat AI: deployment tutorial

This tutorial explains how NVIDIA NeMo Guardrails fits into a Red Hat AI deployment. The presenter describes a guardrails server that inspects prompts and responses, with input detectors running before requests reach the model and output detectors checking responses before users receive them.

An administrator defines a Kubernetes custom resource, and the TrustyAI Operator deploys the server in front of a vLLM model. Applications send requests to the guardrails endpoint. The backend example uses a Nemotron model through an OpenAI-compatible endpoint inside an OpenShift cluster, a configuration relevant to self-hosted inference.

A pizza-ordering AI agent demonstrates the controls. Without guardrails, the application already rejects some unrelated and sensitive requests. After activation, a request for a Social Security number triggers an explicit detector message. The presenter argues that this separate policy layer reduces dependence on the model's own refusal behavior; the demonstration does not establish protection against every bypass.

The ConfigMap example defines separate input and output regex policies alongside sensitive data detection. Input checks include credit card numbers, Social Security numbers, email addresses and phone numbers; output entity checks cover credit card and Social Security numbers. Execution flows run sensitive data detection before regex checks. The presenter also stresses that guardrails belong alongside evaluation, access controls, testing and ongoing monitoring.