SHAP

SHAP is a Python library for data scientists and machine learning researchers who need to understand how input features contribute to a model's predictions. It uses Shapley values from game theory to assign contributions to individual features. The library is open source under the MIT license and runs in your own Python environment.

Its choice of explanation methods depends on the model you're examining. TreeExplainer computes exact contributions for trees and tree ensembles, with fast C++ implementations for XGBoost, LightGBM, CatBoost, scikit-learn and pyspark models. GPU-accelerated Tree SHAP requires a source build with the CUDA toolkit and the SHAP_ENABLE_CUDA option. For tree models, SHAP also calculates pairwise feature interactions, separating each feature's main effect from effects that depend on another feature.

For neural networks, DeepExplainer uses an approximation related to DeepLIFT, while GradientExplainer uses expected gradients and different approximation assumptions. Examples cover Keras image classifiers and LSTM sentiment models, as well as explanations of an internal layer in VGG16.

LinearExplainer can calculate exact values for linear models with independent features or account for feature correlation through an estimated covariance matrix. KernelExplainer works across model types without assuming a particular architecture, though it's slower than the specialized methods. The package also includes visualizations of feature contributions and notebooks covering tabular predictions, image classification and text sentiment analysis.