Captum

Captum is a Python library for examining what contributes to a PyTorch model's predictions, including input features, learned concepts and training examples. It's for model developers investigating unexpected results, application engineers explaining predictions, and researchers comparing interpretability methods. It works with most PyTorch model types with little modification to the model itself.

Its methods cover different levels of explanation. Integrated Gradients and saliency maps attribute predictions to input features, while neuron and layer methods examine which parts of a network matter for an output. SmoothGrad and VarGrad provide further ways to study feature attribution. For questions beyond individual inputs, Testing with Concept Activation Vectors (TCAV) examines concept influence, and TracIn estimates the influence of training examples.

Captum supports vision, text and other modalities, with integration for models built using torchvision and torchtext. It also includes adversarial attacks and small input perturbations for studying how predictions change. Those capabilities support counterfactual explanations as well as adversarial testing.

Researchers can implement their own interpretability algorithms and benchmark them against methods in the library. Captum is open source under the BSD-3-Clause license and works within Python and PyTorch projects. Application engineers can use its explanations to investigate production model behavior or explain a specific result, such as a movie recommendation.