Introduction Copy Section

---
id: introduction
title: Introduction
---
Captum (“comprehension” in Latin) is an open source, extensible library for model interpretability built on PyTorch.

With the increase in model complexity and the resulting lack of transparency, model interpretability methods have become increasingly important. Model understanding is both an active area of research as well as an area of focus for practical applications across industries using machine learning. Captum provides state-of-the-art algorithms, including Integrated Gradients, to provide researchers and developers with an easy way to understand which features are contributing to a model’s output.

Captum helps ML researchers more easily implement interpretability algorithms that can interact with PyTorch models. It also allows researchers to quickly benchmark their work against other existing algorithms available in the library.

For model developers, Captum can be used to improve and troubleshoot models by facilitating the identification of different features that contribute to a model’s output in order to design better models and troubleshoot unexpected model outputs.

Target Audience Copy Chapter

The primary audiences for Captum are model developers who are looking to improve their models and understand which features are important and interpretability researchers focused on identifying algorithms that can better interpret many types of models.

Captum can also be used by application engineers who are using trained models in production. Captum provides easier troubleshooting through improved model interpretability, and the potential for delivering better explanations to end users on why they’re seeing a specific piece of content, such as a movie recommendation.

---

Overview Copy Section

---
id: overview
title: Overview
---

This overview describes the basic components of Captum and how they work
together. For a high-level view of what Captum tries to achieve in more
abstract terms, please see the Introduction.

TODO: Add content here

---

README Copy Section

This directory contains the source files for Captum's Docusaurus documentation.
See the repo's README for additional information.

---

Algorithms Comparison Matrix Copy Section

---
id: algorithms_comparison_matrix
title: Algorithm Comparison Matrix
---

Attribution Algorithm Comparison Matrix

Please, scroll to the right for more details.
<table style="overflow-x: scroll; overflow: auto; display: block;" width="100%">
<tr>
<th style="padding: 30px;">Algorithm</th>
<th style="padding: 30px;">Type</th>
<th style="padding: 80px 100px;">Application</th>
<th style="padding: 30px;">Space&nbsp;Complexity</th>
<th style="padding: 30px;">Model&nbsp;Passes&nbsp;(Forward Only or Forward and Backward))</th>
<th style="padding: 30px;">Number&nbsp;of&nbsp;Samples&nbsp;Passed through Model's Forward (and Backward) Passes</th>
<th style="padding: 30px;">Requires&nbsp;Baseline&nbsp;aka Reference ?</th>
<th style="padding: 80px 150px;">Description</th>
</tr>
<tr>
<td><strong>Integrated Gradients˚^</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function.</td>
<td>O(#steps #examples #features)</td>
<td>Forward and Backward</td>
<td>#steps * #examples</td>
<td>Yes (Single Baseline Per Input Example)</td>
<td>Approximates the integral of gradients along the path (straight line from baseline to input) sand multiplies with (input - baseline)</td>
</tr>
<tr>
<td><strong>DeepLift˚^</strong></td>
<td>Application</td>
<td>Any model that can be represented as a differentiable function. NOTE: In our implementation we perform gradient overrides only for a small set of non-linearities. If your model has any kind of special non-linearities that aren't included in our list, we need to add that support separately. </td>
<td>O(#examples * #features)</td>
<td>Forward and Backward</td>
<td>#examples</td>
<td>Yes (Single Baseline Per Input Example)</td>
<td>Explains differences in the non-linear activations' outputs in terms of the differences of the input from its corresponding reference. NOTE: Currently, only rescale rule is supported.</td>
</tr>
<tr>
<td><strong>DeepLiftSHAP˚^</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function. NOTE: In our implementation we perform gradient overrides only for a small set of non-linearities. If your model has any kind of special non-linearities that aren't included in our list, we need to add that support separately.</td>
<td>O(#examples #features #baselines)</td>
<td>Forward and Backward</td>
<td>#steps * #examples</td>
<td>Yes (Multiple Baselines Per Input Example)</td>
<td> An extension of DeepLift that approximates SHAP values. For each input example it considers a distribution of baselines and computes the expected value of the attributions based on DeepLift algorithm across all input-baseline pairs. NOTE: Currently, only rescale rule is supported. </td>
</tr>
<tr>
<td><strong>GradientSHAP˚^</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function.</td>
<td>O(#examples # samples #features + #baselines * #features)</td>
<td>Forward and Backward</td>
<td>#examples * #samples</td>
<td>Yes (Multiple Baselines Per Input Example)</td>
<td> Approximates SHAP values based on the expected gradients. It adds gaussian noise to each input example #samples times, selects a random point between each sample and randomly drawn baseline from baselines' distribution, computes the gradient for it and multiples it with (input - baseline). Final SHAP values represent the expected values of gradients * (input - baseline) for each input example.</td>
</tr>
<tr>
<td><strong>Input * Gradient</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function.</td>
<td>O(#examples * #features)</td>
<td>Forward and Backward</td>
<td>#examples </td>
<td>No</td>
<td>Multiplies model inputs with the gradients of the model outputs w.r.t. those inputs.</td>
</tr>
<tr>
<td><strong>Saliency˚</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function.</td>
<td>O(#examples * #features)</td>
<td>Forward and Backward</td>
<td>#examples </td>
<td>No</td>
<td>The gradients of the output w.r.t. inputs.</td>
</tr>
<tr>
<td><strong>Guided BackProp˚ / DeconvNet˚</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function. NOTE: this algorithm makes sense to use if the model contains RELUs since it is based on the idea of overriding the gradients of inputs or outputs of any ReLU.</td>
<td>O(#examples * #features)</td>
<td>Forward and Backward</td>
<td>#examples </td>
<td>No</td>
<td>Computes the gradients of the model outputs w.r.t. its inputs. If there are any RELUs present in the model, their gradients will be overridden so that only positive gradients of the inputs (in case of Guided BackProp) and outputs (in case of deconvnet) are back-propagated.</td>
</tr>
<tr>
<td><strong>Guided GradCam</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function. NOTE: this algorithm is designed primarily for CNNs.</td>
<td>O(2 #examples #features)</td>
<td>Forward and Backward</td>
<td>#examples </td>
<td>No</td>
<td>Computes the element-wise product of Guided BackProp and up-sampled positive GradCam attributions.</td>
</tr>
<tr>
<td><strong>LayerGradCam</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function and has a convolutional layer. NOTE: this algorithm is designed primarily for CNNs.</td>
<td>O(#examples * #features)</td>
<td>Forward and Backward</td>
<td>#examples </td>
<td>No</td>
<td>Computes the gradients of model outputs w.r.t. selected input layer, averages them for each output channel and multiplies with the layer activations.</td>
</tr>
<tr>
<td><strong>Layer Internal Influence</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function and has a convolutional layer. NOTE: this algorithm is designed primarily for CNNs.</td>
<td>O(#steps #examples #features)</td>
<td>Forward and Backward</td>
<td>#steps * #examples </td>
<td>Yes (Single Baseline Per Input Example)</td>
<td>Approximates the integral of gradients along the path from baseline to inputs for selected input layer. </td>
</tr>
<tr>
<td><strong>Layer Conductance˚</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function and has a convolutional layer.</td>
<td>O(#steps #examples #features)</td>
<td>Forward and Backward</td>
<td>#steps * #examples </td>
<td>Yes (Single Baseline Per Input Example)</td>
<td>Decomposes integrated gradients via chain rule. It approximates the integral of gradients defined by a chain rule, described as the gradients of the output w.r.t. to the neurons multiplied by the gradients of the neurons w.r.t. the inputs, along the path from baseline to inputs. Finally, the latter is multiplied by (input - baseline).</td>
</tr>
<tr>
<td><strong>Layer Gradient * Activation</strong></td>
<td>Gradient</td>
<td>Any model that can be represented as a differentiable function and has a convolutional layer.</td>
<td>O(#examples * #features)</td>
<td>Forward and Backward</td>
<td>#examples </td>
<td>No</td>
<td>Computes element-wise product of layer activations and the gradient of the output w.r.t. that layer.</td>
</tr>
<tr>
<td><strong>Layer Activation</strong></td>
<td> - </td>
<td>Any neural network model. </td>
<td>O(#examples * #features)</td>
<td>Forward and Backward</td>
<td>#examples </td>
<td>No</td>
<td>Computes the inputs or outputs of selected layer.</td>
</tr>
<tr>
<td><strong>Feature Ablation˚^</strong></td>
<td> Perturbation </td>
<td>Any traditional or neural network model. </td>
<td>O(#examples #features #perturbations_per_eval) </td>
<td>Forward</td>
<td>#examples * #features </td>
<td>Yes (Single Baseline Per Input Example; Usually, zero baseline is used)</td>
<td>Assigns an importance score to each input feature based on the magnitude changes in model output or loss when those features are replaced by a baseline (usually zeros) based on an input feature mask.</td>
</tr>
<tr>
<td><strong>Feature Permutation</strong></td>
<td> Perturbation </td>
<td>Any traditional or neural network model. </td>
<td>O(#examples #features #perturbations_per_eval)</td>
<td>Forward</td>
<td>#examples * #features </td>
<td>No (Internally in our implementation permuted features for each batch are treated as baselines)</td>
<td>Assigns an importance score to each input feature based on the magnitude changes in model output or loss when those features are permuted based on input feature mask. </td>
</tr>
<tr>
<td><strong>Occlusion</strong></td>
<td> Perturbation </td>
<td> Any traditional or neural network model. NOTE: this algorithm has been primarily used for computer vision but could theoretically also be used for other applications as well. In addition to that this algorithm also requires strides which indicates the length of the steps required for sliding k-dimensional window.</td>
<td>O(#examples #features #ablations_per_eval * 1 / #strides)</td>
<td>Forward</td>
<td>#examples * #features </td>
<td>Yes (usually, zero baseline is used)</td>
<td>Assigns an importance score to each input feature based on the magnitude changes in model output when those features are replaced by a baseline (usually zeros) using rectangular sliding windows and sliding strides. If a features is located in multiple hyper-rectangles the importance scores are averaged across those hyper-rectangles.</td>
</tr>
<tr>
<td><strong>Shapely Value</strong></td>
<td>Perturbation </td>
<td>Any traditional or neural network model.</td>
<td>O(#examples #features #perturbations_per_eval )</td>
<td>Forward</td>
<td>#examples #features #features! </td>
<td>Yes (usually, zero baseline is used)</td>
<td>Computes feature importances based on all permutations of all input features. It adds each feature for each permutation one-by-one to the baseline and computes the magnitudes of output changes for each feature which are ultimately being averaged across all permutations to estimate final attribution score. </td>
</tr>
<tr>
<td><strong>Shapely Value Sampling</strong></td>
<td>Perturbation </td>
<td>Any traditional or neural network model.</td>
<td>O(#examples #features #perturbations_per_eval )</td>
<td>Forward</td>
<td>#examples #features #samples</td>
<td>Yes (usually, zero baseline is used)</td>
<td>Similar to Shapely value, but instead of considering all feature permutations it considers only #samples random permutations.</td>
</tr>
<tr>
<td><strong>NoiseTunnel</strong></td>
<td> - </td>
<td>This can be used in combination with any above mentioned attribution algorithms</td>
<td>Depends on the choice of above mentioned attribution algorithm. </td>
<td>Forward or Forward and Backward - It depends on the choice of above mentioned attribution algorithm.</td>
<td>#examples #features #samples</td>
<td>Depends on the choice of above mentioned attribution algorithm. </td>
<td>Depends on the choice of above mentioned attribution algorithm. | Adds gaussian noise to each input example #samples times, calls any above mentioned attribution algorithm for all #samples per example and aggregates / smoothens them based on different techniques for each input example. Supported smoothing techniques include: smoothgrad, vargrad, smoothgrad_sq.</td>
</tr>

</table>

^ Including Layer Variant

˚ Including Neuron Variant

<a href="/img/algorithms_comparison_matrix.png">Algorithm Comparison Matrix.png</a>

---

Attribution Algorithms Copy Section

---
id: attribution_algorithms
title: Algorithm Descriptions
---

Captum is a library within which different interpretability methods can be implemented. The Captum team welcomes any contributions in the form of algorithms, methods or library extensions!

The attribution algorithms in Captum are separated into three groups, primary attribution, layer attribution and neuron attribution, which are defined as follows:
* Primary Attribution: Evaluates contribution of each input feature to the output of a model.
* Layer Attribution: Evaluates contribution of each neuron in a given layer to the output of the model.
* Neuron Attribution: Evaluates contribution of each input feature on the activation of a particular hidden neuron.

Below is a short summary of the various methods currently implemented for primary, layer, and neuron attribution within Captum, as well as noise tunnel, which can be used to smooth the results of any attribution method.

Beside attribution algorithms Captum also offers metrics to estimate the trustworthiness of model explanations.
Currently we offer infidelity and sensitivity metrics that help us to estimate the goodness of explanations.

Primary Attribution Copy Chapter

Integrated Gradients Copy Section

Integrated Gradients along the i - th dimension of input X. Alpha is the scaling coefficient. The equations are copied from the original paper.

The cornerstones of this approach are two fundamental axioms, namely sensitivity and implementation invariance. More information regarding these axioms can be found in the original paper.

To learn more about Integrated Gradients, visit the following resources:
- Original paper

Gradient SHAP Copy Section

The computed attributions approximate SHAP values under the assumptions that the input features are independent and that the explanation model is linear between the inputs and given baselines.

To learn more about GradientSHAP, visit the following resources:
- SHAP paper
- Original Implementation

DeepLIFT Copy Section

x is the input neuron with a difference from reference Δx, and t is the target neuron with a difference from reference Δt. C is then the contribution of Δx to Δt.

Like partial derivatives (gradients) used in back propagation, multipliers obey the Chain Rule. According to the formulations proposed in this paper. DeepLIFT can be overwritten as the modified partial derivatives of output of non-linear activations with respect to their inputs.

Currently, we only support Rescale Rule of DeepLIFT Algorithms. RevealCancel Rule will be implemented in later releases.

To learn more about DeepLIFT, visit the following resources:
- Original paper
- Explanatory videos attached to paper
- Towards Better Understanding of Gradient-Based Attribution Methods for Deep Neural Networks

DeepLIFT SHAP Copy Section

DeepLIFT's rules for non-linearities serve to linearize non-linear functions of the network, and the method approximates SHAP values for the linearized version of the network. The method also assumes that the input features are independent.

To learn more about DeepLIFT SHAP, visit the following resources:
- SHAP paper

Saliency Copy Section

To learn more about Saliency, visit the following resources:
- Original paper

Input X Gradient Copy Section

Guided Backpropagation and Deconvolution Copy Section

To learn more about Guided Backpropagation, visit the following resources:
- Original paper

To learn more about Deconvolution, visit the following resources:
- Original paper
- Salient Deconvolutional Networks

Guided GradCAM Copy Section

Guided GradCAM was proposed by the authors of GradCAM as a method to combine the high-resolution nature of Guided Backpropagation with the class-discriminative advantages of GradCAM, which has lower resolution due to upsampling from a convolutional layer.

To learn more about Guided GradCAM, visit the following resources:
- Original paper
- Website

Feature Ablation Copy Section

Feature Permutation Copy Section

To learn more about Feature Permutation, visit the following resources:
- Interpretable ML Book

Occlusion Copy Section

To learn more about Occlusion (also called grey-box / sliding window method), visit the following resources:
- Original paper
- DeepExplain Implementation

Shapley Value Sampling Copy Section

Since this method is extremely computationally intensive for larger numbers of features, we also implement Shapley Value Sampling, where we sample some random permutations and average the marginal contribution of features based on these permutations.
Like feature ablation, input features can also be grouped and added together rather than individually.

To learn more about Shapley Value Sampling, visit the following resources:
- Original paper
- Interpretable ML Book

Lime Copy Section

We offer two implementation variants of this method, LimeBase and Lime.
LimeBase provides a generic framework to train a surrogate interpretable model. This differs from most API of other attribution methods, since the method returns a representation of the interpretable model (e.g. coefficients of the linear model). On the other hand, Lime provides a more specific implementation than LimeBase in order to expose a consistent API with other perturbation-based algorithms.

To learn more about Lime, visit the following resources:
- Original paper

KernelSHAP Copy Section

To learn more about KernelSHAP, visit the following resources:
- Original paper

Layer Attribution Copy Chapter

Layer Conductance Copy Section

Conductance builds on Integrated Gradients (IG) by looking at the flow of IG attribution which occurs through the hidden neuron. The formal definition of total conductance of a hidden neuron y (from the original paper) is as follows:

For more efficient computation of layer conductance, we use the idea presented in this paper to avoid computing the gradient of each neuron with respect to the input.

To learn more about Conductance, visit the following resources:
- Original Paper
- Computationally Efficient Measures of Internal Neuron Importance

Internal Influence Copy Section

To learn more about Internal Influence, visit the following resources:
- Original Paper

Layer Activation Copy Section

Layer Gradient X Activation Copy Section

GradCAM Copy Section

GradCAM is a layer attribution method designed for convolutional neural networks, and is usually applied to the last convolutional layer.
GradCAM computes the gradients of the target output with respect to the given layer, averages for each output channel (dimension 2 of output), and multiplies the average gradient for each channel by the
layer activations. The results are summed over all channels and a ReLU is applied to the output, returning only non-negative attributions.

This procedure sums over the second dimension (# of channels), so the output of GradCAM attributions will have a second dimension of 1, but all other dimensions will match that of the layer output.

Although GradCAM directly attributes the importance of different neurons in the target layer, GradCAM is often used as a general attribution method. To accomplish this, GradCAM attributions are upsampled and viewed as a mask to the input, since a convolutional layer output generally matches the input image spatially.

To learn more about GradCAM, visit the following resources:
- Original paper
- Website

Layer Integrated Gradients Copy Section

To learn more about Integrated Gradients, see this section above.

Layer GradientSHAP Copy Section

To learn more about Gradient SHAP, see this section above.

Layer DeepLIFT Copy Section

To learn more about DeepLIFT, see this section above.

Layer DeepLIFT SHAP Copy Section

Layer DeepLIFT SHAP is the analog of DeepLIFT SHAP for a particular layer. Layer DeepLIFT SHAP takes a distribution of baselines and computes the Layer DeepLIFT attribution for each input-baseline pair and averages the resulting attributions per input example.

To learn more about DeepLIFT SHAP, see this section above.

Layer Feature Ablation Copy Section

Neuron Attribution Copy Chapter

Neuron Conductance Copy Section

Conductance for a particular neuron builds on Integrated Gradients (IG) by looking at the flow of IG attribution from each input through the particular neuron. The formal definition of conductance of neuron y for the attribution of input i (from the original paper) is as follows:

Note that based on this definition, summing the neuron conductance (over all input features) always equals the layer conductance for the particular neuron.

To learn more about Conductance, visit the following resources:
- Original Paper
- Computationally Efficient Measures of Internal Neuron Importance

Neuron Gradient Copy Section

Neuron Integrated Gradients Copy Section

To learn more about Integrated Gradients, see this section above.

Neuron Guided Backpropagation and Deconvolution Copy Section

To learn more about Guided Backpropagation and Deconvolution, see this section above.

Neuron GradientSHAP Copy Section

To learn more about GradientSHAP, see this section above.

Neuron DeepLIFT Copy Section

To learn more about DeepLIFT, see this section above.

Neuron DeepLIFT SHAP Copy Section

Neuron DeepLIFT SHAP is the analog of DeepLIFT SHAP for a particular neuron. Neuron DeepLIFT SHAP takes a distribution of baselines and computes the Neuron DeepLIFT attribution for each input-baseline pair and averages the resulting attributions per input example.

To learn more about DeepLIFT SHAP, see this section above.

Neuron Feature Ablation Copy Section

Noise Tunnel Copy Chapter

To learn more about Noise Tunnel methods, visit the following resources:
- SmoothGrad Original paper
- VarGrad Original paper

Metrics Copy Chapter

Infidelity Copy Section

It is derived from the completeness property of well-known attribution algorithms, such as Integrated Gradients, and is a computationally more efficient and generalized notion of Sensitivy-n. The latter measures correlations between the sum of the attributions and the differences of the predictor function at its input and fixed baseline. More details about the Sensitivity-n can be found here:
https://arxiv.org/abs/1711.06104
More details about infidelity measure can be found here:
- Original paper

Sensitivity Copy Section

In order to approximate sensitivity measure, by default, we sample from a sub-space of an L-Infinity ball with a default radius.
The users can modify both the radius of the ball and the sampling function.
More details about sensitivity measure can be found here:
- Original paper

---

Contribution Guide Copy Section

---
id: contribution_guidelines
title: The Captum Contribution Process
---

The Captum development process involves a healthy amount of open discussions between the core development team and the community.
Captum operates similarly to most open source projects on GitHub. However, if you've never contributed to an open source project before, here is the basic process.

1. Figure out what you're going to work on.
2. Figure out the scope of your change and reach out for design comments or feedback on a GitHub issue.
* If you want to contribute a new feature or algorithm, please, check out the section on Proposing new features and algorithms.
* We expect that all contributed features, algorithms and improvements follow these guidelines, analogous to existing methods in Captum:
* Algorithm APIs should maintain as much similarity as possible with existing APIs to ensure ease-of-use when switching between algorithms.
* Detailed documentation including a summary of the general algorithm, example usage, and descriptions of all parameters and returns as well as documentation of any limitations.
* Explicit test cases for test models, similar to existing method test cases.
* Test the algorithms and changes on real models and real datasets. Use the datasets and models mentioned in the benchmarking section or in Captum tutorials or propose new datasets in GitHub issue’s section and we will review them.
* Support and tests for model wrappers such as DataParallel, DistributedDataParallel, and JIT.
* Type hints and formatting as described in the general contributing guidelines.
3. Code it out!
4. Open a Pull Request!
5. Iterate on the pull request until it's accepted!

Proposing New Features and Algorithms Copy Chapter

New feature ideas are best discussed on a specific issue. Please include as much information as you can, including any accompanying data and your proposed solution. The Captum team and community frequently reviews new issues and comments where they think they can help.

While we would like to accept as many algorithms and features to Captum as possible, the core team is responsible for maintaining and supporting all functionalities in the future, so we need to ensure the package remains maintainable. Additionally, we also want to ensure that the supported methods cover the variety and breadth of methods in model interpretability, without overwhelming new users with many new or experimental methods. To balance these objectives, we have established guidelines and general criteria to help make decisions when considering new algorithms and features. The evaluation of any model interpretability algorithm can be subjective, especially since there are no general purpose qualitative and quantitative metrics measuring their quality. Hence, we provide these guidelines as a starting point and will utilize discussions in GitHub issues (https://github.com/pytorch/captum/issues) to ensure a transparent process while reviewing proposals.

Before contributing an algorithm to Captum, please review these guidelines and provide as much context as possible regarding which criteria apply to the proposed method.

1. Usage / Citations and Impact on Model Interpretability Community
* A primary factor we look at when considering new methods is popularity of the method and impact on the model interpretability community. Our baseline for consideration is generally 20 citations or 100 forks on GitHub, but this is not an absolute requirement. If an algorithm is newly published or has had less visibility in the community, this criteria may not be satisfied, but strengths in the remaining criteria could justify acceptance.
2. Multimodality
* Since one of the core values of Captum is multimodality, we prefer algorithms that are generic so that they can be used for different types of model architectures and input types. If the algorithm is unimodal, let’s say, if it works only for text or vision models, it is important to clarify what type of vision or text models the implementation supports. Does it work for LSTMs only? In other words, it is important to discuss the scope of the algorithm and its impact.
3. Benchmarking Results
* We also would like comparisons with existing algorithm benchmarks in terms of performance and visual interpretation, and strong results compared to baselines are a plus when considering inclusion in Captum.
* Performance benchmarking
* Please report runtime execution numbers of the algorithm in CPU and GPU environments. Describe the environment where the experiments were conducted. It is also encouraged to do performance comparison with existing baseline approaches and report those numbers as well.
* Visual interpretation
* Although visual interpretations can be deceptive, it is important to compare newly implemented algorithms with other state of the art approaches side by side using well-known baseline models and datasets. For baseline models and datasets check out the section on Algorithm benchmarking on real datasets and models section
* In addition to visual interpretations, if possible, for attribution algorithms we can also assess infidelity and sensitivity metrics provided in captum.metrics package.
4. Axiomatic and Mathematically Sound
* Since evaluation and qualitative analysis of interpretability methods can sometimes be misleading, methods that are axiomatic or have strong theoretical justification are preferred.

If you think that proposed algorithm/feature satisfies many of the above criteria, please open an issue on GitHub (https://github.com/pytorch/captum/issues) to discuss the method with the core Captum team members before submitting a pull request. If the method seems suitable for inclusion in Captum, we will generally request a design document or proposal, explaining the proposed API and structure for the new algorithm. An example of a design proposal for LIME and Kernel SHAP can be found here (https://github.com/pytorch/captum/issues/467).

If an algorithm or feature adds only marginal improvements or does not meet most the criteria described above, then we would suggest including it into our AWESOME_LIST.MD (https://github.com/pytorch/captum/blob/master/AWESOME_LIST.md) instead of adding it to the core Captum library. In the future, if the algorithm gains more popularity and acceptance in the model interpretability community, we would be happy to accept a PR to add it to the Captum core library.

Note that we reserve the right to decide not to include any algorithms that meet the above criteria, but we are unable to support in the long run.

Algorithm benchmarking on real datasets and models Copy Chapter

NLP
- We provide a sample CNN-based model for sensitivity analysis. Although, currently, we do not provide a LSTM model in the tutorials, we strongly encourage you to test the model on a baseline LSTM model (e.g. the original LSTM model described inLong short-term memory (https://dl.acm.org/doi/10.1162/neco.1997.9.8.1735) for sentiment analysis) as well.
Besides that, it is encouraged to test the algorithms on Bert models as well. An example Bert Q&A model is available in the tutorial’s section.
https://captum.ai/tutorials/Bert_SQUAD_Interpret
https://captum.ai/tutorials/IMDB_TorchText_Interpret

Vision
- We provide a sample toy model for the CIFAR dataset and examples with a ResNet model.
https://captum.ai/tutorials/CIFAR_TorchVision_Interpret
https://captum.ai/tutorials/Resnet_TorchVision_Interpret
These would be great starting points for benchmarking.
We also encourage you to test your models on other well-known benchmarks such as MNIST digit and fashion
dataset, Inception and VGG models.

Baseline MLP Classification Models
- In terms of baseline MLP models and datasets we encourage you to use titanic dataset and the simple MLP model that we built in the following tutorial:
https://captum.ai/tutorials/Titanic_Basic_Interpret

Baseline Regression models
- Boston House prices dataset and model can be found here:
https://captum.ai/tutorials/House_Prices_Regression_Interpret

Multimodal
- You can use VQA model and dataset described here:
https://captum.ai/tutorials/Multimodal_VQA_Captum_Insights

Docstring style Copy Chapter

Docstring is required for all public APIs to provide users the details of the arguments and returns. Our API documentation is generated from the docstring. Captum adopts a customized docstring format modified on top of Google style. Specifically, each argument should be listed as arg_name (type): description in the Args: section. The argument typing convention:
- primitive types: int, str, float, bool
- common collection types: list, tuple, dict
- PEP 585 has deprecated the duplicate types: List, Tuple, Dict
- element types: list[int], dict[int, str]
- other foundamental types: Any, Callable, Iterable
- class types: MyClass, external_lib.SomeClass
- omit torch for common Pytorch types: Tensor, nn.Module
- use or and , for union types: type1 or type2, type1, tyep2, or type3
- PEP 604 proposes to use | to connect types: type1 | type2. We may consider migration later.
- append optional for argument with default value: int, optional
- append default value to the end of the description: Default: None
- Notice this is different with python's type hint Optional[...], which indicate if the argument can be None

---

Faq Copy Section

---
id: faq
title: FAQ
---

* How do I set the target parameter to an attribution method?
* I am facing Out-Of-Memory (OOM) errors when using Captum. How do I resolve this?
* I am using a perturbation based method, and attributions are taking too long to compute. How can I speed it up?
* Are SmoothGrad or VarGrad supported in Captum?
* How do I use Captum with BERT models?
* My model inputs or outputs token indices, and when using Captum I see errors relating to gradients, how do I resolve this?
* Can my model use functional non-linearities (E.g. nn.functional.ReLU) or can reused modules be used with Captum?
* Do JIT models, DataParallel models, or DistributedDataParallel models work with Captum?
* I am working on a new interpretability or attribution method and would like to add it to Captum. How do I proceed?
* I am using a gradient-based attribution algorithm such as integrated gradients for a RNN or LSTM network and I see 'cudnn RNN backward can only be called in training mode'. How can I resolve this issue ?

How do I set the target parameter to an attribution method? Copy Section

The purpose of target is to select a single (scalar) value for each example in the output of your model to compute attributions based on the given target parameter. Each attribution method answers the question of how important each input value is towards a particular output scalar value.

If the model only returns a scalar value per example, which is common in either a regression case or binary classification, then you don’t need to pass the target argument or simply set it None.

If the model has a 2D output, which is common with classification cases, then you must pass a target to identify which output value to compute attributions for. Consider an output with the shape N x 3, with N being the number of examples and 3 being the number of classes. The purpose of targets is to select one of the 3 values for each example. You could then pass targets as just a single integer 0 or 1 or 2, which corresponds to attributing to the output for class 0, 1, or 2 respectively for all examples. If you are passing a batch of multiple examples (e.g. N = 4), you can also pass a different target for each example, by providing targets as a list or tensor, e.g. [0, 1, 0, 0] or torch.tensor([0, 1, 0, 0]), which would return attributions for class 0 for the first, third and fourth input examples (how important each input value was for the prediction of class 0), and class 1 for the second one (how important each input value was for the prediction of class 1).

If your model has > 2D output, then you can pass a tuple containing the indices of the particular scalar value in the output tensor for which attributions are desired. For instance, with an output shape of N x 3 x 4 x 5, target should be a tuple such as (2, 3, 2), corresponding to the index for each dimension other than the first. Like the 2D case, a list of tuples can be provided for a batch of input examples. Note that in some cases with > 2D output it may be beneficial to attribute with respect to the sum of particular output values rather than each value independently. The attribution methods would then answer the question of how important each input value is to the sum of the chosen values. To accomplish this, it is necessary to wrap your function in a function which sums the desired output values and provide the wrapper as the forward_func for the attribution method.

I am facing Out-Of-Memory (OOM) errors when using Captum. How do I resolve this? Copy Section

When using methods such as Integrated Gradients, Conductance, Internal Influence or other algorithms with n_steps argument, the input is expanded n_steps times, which often becomes too large to fit in memory. To address this issue, you can either reduce n_steps, which may lead to lower-quality approximations, or use the internal_batch_size argument, which allows dividing the expanded input into batches which are processed sequentially. Note that using internal_batch_size will increase runtime since it runs multiple evaluations, so it is best to utilize the largest feasible batch size within memory constraints.

If you are using a perturbation-based attribution algorithm, try reducing perturbations_per_eval if it is set to a value greater than 1. This will limit the number of perturbations processed in one batch.

For other algorithms, it might be necessary to try a smaller input batch size.

I am using a perturbation based method, and attributions are taking too long to compute. How can I speed it up? Copy Section

For perturbation based methods, including Feature Ablation and Occlusion, each perturbation of the input batch is processed sequentially by default. If enough memory is available or the input batch size is small, it is often possible to process multiple perturbations in one batch, which can substantially speed up the performance of these algorithms. To enable this, simply set the perturbations_per_eval argument to the desired value.

If you have multiple GPUs machines available, you can also wrap your model with DataParallel or look into torch.distributed / DistributedDataParallel, these PyTorch features work with all Captum methods.

Are SmoothGrad or VarGrad supported in Captum? Copy Section

Yes! SmoothGrad and VarGrad are available through NoiseTunnel in Captum, which can be used with any attribution algorithm in Captum. More details on Noise Tunnel can be found in the documentation here.

How do I use Captum with BERT models? Copy Section

We have a tutorial demonstrating usage of Integrated Gradients on BERT here.

My model inputs or outputs token indices, and when using Captum I see errors relating to gradients, how do I resolve this? Copy Section

For NLP models that take token indices as inputs, we cannot take gradients with respect to indices. To apply gradient-based attribution methods, it is necessary to replace the embedding layer with an InterpretableEmbedding layer or use LayerIntegratedGradients to compute attribution with respect to the embedding output. Attribution can then be summed for all dimensions of the embedding to evaluate importance of each token / index. For examples of this process, take a look at the IMDB or BERT tutorials.

If the output of the model is a token index, such as an image captioning cases, it is necessary to attribute with respect to the token score or probability rather than the index. Make sure that the model returns this and use target to choose the appropriate scalar score to attribute with respect to.

Can my model use functional non-linearities (E.g. nn.functional.ReLU) or can reused modules be used with Captum? Copy Section

Most methods will work fine with functional non-linearities and arbitrary operations. Some methods, which require placing hooks during back-propagation, including DeepLift, DeepLiftShap, Guided Backpropagation, and Deconvolution will not work appropriately with functional non-linearities and must use the corresponding module activation (e.g. torch.nn.ReLU) which should be initialized in the module constructor. For DeepLift, it is important to also not reuse modules in the forward function, since this can cause issues in the propagation of multipliers. Computing layer or neuron attribution with layer modules that are used multiple times generally computes attributions for the last execution of the module. For more information regarding these restrictions, refer to the API documentation for the specific method, including DeepLift, DeepLiftShap, Guided Backpropagation, and Deconvolution.

Do JIT models, DataParallel models, or DistributedDataParallel models work with Captum? Copy Section

Yes, we have support for all these model types. Note that JIT models do not yet support hooks, so any methods using hooks including layer and neuron attribution methods, DeepLift, Guided Backprop, and Deconvolution are not supported. DataParallel and DistributedDataParallel are supported with all model types.

I am working on a new interpretability or attribution method and would like to add it to Captum. How do I proceed? Copy Section

For interpretability methods created by the community, we have two methods of involvement:

1. Awesome List - We maintain a list of interesting external projects that focus on interpretability that may be useful for users looking for functionality beyond what’s available in Captum.
2. Inclusion in Captum - New attribution algorithms that fit the structure of Captum can be considered for contribution to the contrib package of algorithms in Captum. We review proposals for new additions to the contrib package on a case-by-case basis and consider factors such as publication history, quantitative and qualitative evaluation, citations, etc.

We are still working out the logistics of setting these up and will update this with more information once it’s available.

How can I resolve cudnn RNN backward error for RNN or LSTM network? Copy Section

---

Getting Started Copy Section

---
id: getting_started
title: Getting Started
---

This section shows you how to get up and running with Captum.

Installing Captum Copy Chapter

#### Installation Requirements:

- Python >= 3.6
- PyTorch >= 1.2
- numpy

Captum is easily installed via
Anaconda (recommended)
or pip:

conda install captum -c pytorch

pip install captum

For more detailed installation instructions, please see the
Project Readme
on GitHub.

Tutorials Copy Chapter

We have several tutorials to help get you off the ground with Captum. The tutorials are Jupyter notebooks and cover the basics along with demonstrating usage of Captum with models of different modalities.

View the tutorials page here.

API Reference Copy Chapter

For an in-depth reference of the various Captum internals, see our
API Reference.

Contributing Copy Chapter

You'd like to contribute to Captum? Great! Please see
here
for how to help out.

---

CONTRIBUTING Copy Section

Contributing to Captum

Thank you for your interest in contributing to Captum! We want to make contributing to Captum as easy and transparent as possible.
Before you begin writing code, it is important that you share your intention to contribute with the team, based on the type of contribution:

1. You want to propose and implement a new algorithm, add a new feature or fix a bug. This can be both code and documentation proposals.
1. For all non-outstanding features, bug-fixes and algorithms in the Captum issue list (https://github.com/pytorch/captum/issues) please create an issue first.
2. If the implementation requires API or any other major code changes (new files, packages or algorithms), we will likely request a design document to review and discuss the design and implementation before making changes. An example design document for LIME can be found here (https://github.com/pytorch/captum/issues/467).
3. Once we agree that the plan looks good or confirmed that the change is small enough to not require a detailed design discussion, go ahead and implement it!

2. You want to implement a feature or bug-fix for an outstanding issue.

1. Search for your issue in the Captum issue list (https://github.com/pytorch/captum/issues).
2. Pick an issue and comment that you'd like to work on the feature or bug-fix.
3. If you need more context on a particular issue, please ask and we’ll be happy to help.

Once you implement and test your feature or bug-fix, please submit a Pull Request to https://github.com/pytorch/captum (https://github.com/pytorch/pytorch).

This document covers some of the techical aspects of contributing to Captum. More details on what we are looking for in the contributions can be found in the Contributing Guidelines.

Development installation Copy Chapter

To get the development installation with all the necessary dependencies for
linting, testing, and building the documentation, run the following:

git clone https://github.com/pytorch/captum.git
cd captum
pip install -e .[dev]

Our Development Process Copy Chapter

#### Code Style

Captum uses ufmt and flake8 to
enforce a common code style across the code base. ufmt and flake8 are installed easily via
pip using pip install ufmt flake8, and run locally by calling

ufmt format .
flake8 .

We feel strongly that having a consistent code style is extremely important, so
Github Actions will fail on your PR if it does not adhere to the ufmt or flake8 formatting style.

#### Type Hints

Captum is fully typed using python 3.6+
type hints.
We expect any contributions to also use proper type annotations, and we enforce
consistency of these in our continuous integration tests.

To type check your code locally, install mypy,
which can be done with pip using pip install "mypy>=0.760"
Then run this script from the repository root:

./scripts/run_mypy.sh

#### Unit Tests

To run the unit tests, you can either use pytest (if installed):

pytest -ra

python -m unittest

To get coverage reports we recommend using the pytest-cov plugin:

pytest -ra --cov=. --cov-report term-missing

#### Documentation

Captum's website is also open source, and is part of this very repository (the
code can be found in the website folder).
It is built using Docusaurus, and consists of three
main elements:

1. The documentation in Docusaurus itself (if you know Markdown, you can
already contribute!). This lives in the docs.
2. The API reference, auto-generated from the docstrings using
Sphinx, and embedded into the Docusaurus website.
The sphinx .rst source files for this live in sphinx/source.
3. The Jupyter notebook tutorials, parsed by nbconvert, and embedded into the
Docusaurus website. These live in tutorials.

To build the documentation you will need Node >= 8.x
and Yarn >= 1.5.

The following command will both build the docs and serve the site locally:

./scripts/build_docs.sh

Pull Requests Copy Chapter

1. Fork the repo and create your branch from master.
2. If you have added code that should be tested, add unit tests.
In other words, add unit tests.
3. If you have changed APIs, update the documentation. Make sure the
documentation builds.
4. Ensure the test suite passes.
5. Make sure your code passes both black and flake8 formatting checks.

Issues Copy Chapter

We use GitHub issues to track public bugs. Please ensure your description is
clear and has sufficient instructions to be able to reproduce the issue.

Facebook has a bounty program for the safe
disclosure of security bugs. In those cases, please go through the process
outlined on that page and do not file a public issue.

License Copy Chapter

By contributing to Captum, you agree that your contributions will be licensed
under the LICENSE file in the root directory of this source tree.

---