7.3: Explainable AI (XAI)

Based on SD Sect. 7.5. Explainable AI (XAI) addresses the opacity of complex ML models, enabling scientists and engineers to debug, validate, and deploy trustworthy systems in production.

• Core Concepts

  • Interpretability vs. Explainability

    Interpretability targets intrinsic model transparency (e.g., simulatability, decomposability), which is best for simple models (e.g. decision tree). Explainability generates human-readable justifications for outputs, often post-hoc (after-the-fact) for black-box models like transformers.

  • Local vs. Global Explanations

    Local methods explain single predictions (e.g., per-instance attributions); global ones reveal overall model behavior across datasets (crucial for drift detection).

  • Trade-offs

    Higher accuracy often reduces interpretability. Tips: use surrogates (simple, interpretable models) or approximations (e.g., use LIME or SHAP to create a simplified map) for balance approach. Evaluating fidelity via metrics like explanation accuracy and sufficiency (see below).

• Advanced Techniques

  • Gradient-Based Attribution: it helps us to understand which parts of an input caused an AI to make its final decision.

    E.g. Integrated Gradients: they identify which inputs most influenced an AI’s decision by measuring how the model’s output changes as one “fade in” the input from a blank baseline (starting point) to the final result.

  • SHAP (SHapley Additive exPlanations)

    SHAP uses game theory for consistent additive explanations (or, it uses game theory to fairly divide “credit” among data features). It is based on Shapley values, a concept from cooperative game theory introduced by Lloyd Shapley in 1952.

  • LIME (Local Interpretable Model-agnostic Explanations)

    LIME builds a simple, “interpretable” mini-model (a surrogate) to mimic how the complex AI behaves for specific predictions. It focuses on that local area, testing nearby data points to efficiently figure out which features mattered most in that moment.

  • Attention & Prototypes

    Examine internal model weights to decode reasoning. Attention Maps reveal which tokens or pixels a transformer aggregates during processing, while Prototype-based models compare new inputs directly to definitive, pre-understood examples.

    Use Prototypes, which compare new data to “perfect examples” which AI already understands.

  • Counterfactuals & Influence Functions

    In XAI, spurious correlations happen when an AI model learns misleading, non-causal links (e.g., a dog always having grassin its picture) instead of true features (and causing poor generalization). Approaches to counter them: use techniques from causal inference like Counterfactuals (changing irrelevant features to see if predictions change) or Influence Functions (finding influential training data points) to build robustness.

  • Concept-Based Interpretability: Moves beyond individual pixel/word features to human-scale ideas.

    Concept Bottleneck Models (CBMs): Built intrinsically by forcing a neural network to predict high-level human concepts in an intermediate layer before rendering a final prediction. In other words: design and train the network from scratch with a literal “bottleneck” layer that outputs human concepts before the final answer.

    Linear Probes and TCAV: Post-hoc tools used to inspect standard networks. Linear probes check internal layers to see if human concepts are represented. Testing with Concept Activation Vectors (TCAV) then uses directional derivatives along those probes to quantify how much an abstract concept influences a global model prediction. In other words: apply to standard, already-trained “black-box” models. Linear probes look for concepts hidden in the layers, and TCAV uses calculus to see how important they are.

• Future Directions

  • Shifting from post-hoc interpretations toward hybrid neurosymbolic models (embedding logical rules directly into model design) to provide “explainability by design.”

  • Toward integrating causal XAI, utilizing frameworks (like do-calculus: a set of rules used to determine if a causal effect can be predicted from data by transforming interventional “what-if” queries into probabilistic expressions).

  • Advancements are expected to transform XAI from a transparency tool into a core component of “reasoning-driven intelligence.” This is particularly true for high-stakes sectors (e.g., healthcare, FIN, etc.) where understanding the “why” behind an outcome is legally and operational critical.

1. Mechanistic interpretability (MI)

   MI is a specialized subfield of XAI. While traditional XAI focuses on communicating “why” a model makes a specific prediction, MI seeks to explore the “how” by mapping internal components (e.g., neurons in NN/DL) to human-understandable algorithms.

   Core Concepts

   • Mechanistic vs. Behavioral Interpretability: Mechanistic uncovers causal circuits (e.g., induction heads in transformers); behavioral tests inputs/outputs—prioritize circuits for intervention and scaling laws.

   • Superposition and Polysemanticity: Because models have more “features” than they have physical neurons, they “cram” multiple concepts into the same neuron to save space. This means a single neuron is polysemantic: it activates for many unrelated things at once. Researchers use Sparse Autoencoders (SAEs) to “untangle” this mess. The SAE identifies the distinct, individual concepts (called monosemantic features) that were overlapping, effectively clarifying the purpose of the mid-layer neurons where over 90% of activations are jumbled and multi-purpose.

   • Circuit Discovery: Itis the process of mapping the specific internal “wiring” (subgraphs) an AI model uses to perform a task. Researchers find these subgraphs by using activation patching to swap internal signals between different inputs to see which components are necessary for a correct answer. Similarly, one could use path patching to trace exactly how information flows between specific neurons. To ensure the results are reliable, a common best practice is to only include components that have a significant causal impact and to verify that the discovered circuit still works on out-of-distribution (OOD) data.

   Advanced Techniques

   • Activation Patching: Logit lens + mean ablation to trace circuits; best practice: Patch top-k activations per layer, compute path probabilities with log-additive interventions.

   • Sparse Autoencoders (SAEs): Train an autoencoder with a “sparsity” constraint (e.g., with an L₁ penalty) on the hidden layer. This forces it to activate only a few features at a time for any given input. This makes the features more distinct and interpretable than the network’s original entangled neurons.

   • Transformer Circuits: These are specific network of components (like neurons and attention heads) that work together to perform a single task (e.g. a Previous-Token Head that simply moves information from the immediate past token to the current one). To find these circuits without manual guesswork, researchers use the ACDC (Automated Circuit Discovery) framework, which systematically prunes unimportant connections from the model’s computational graph while monitoring for performance drops. A best practice for efficiency is to prune the pathways that account for less than 5% of the total computation; this typically identifies a sparse sub-graph that still maintains roughly 95% fidelity to the original model’s behavior, effectively isolating the logic of the task from the surrounding noise.

   • Dictionary Learning: This uses SAEs to translate a model’s internal math into a dictionary of clean, human-understandable concepts. To ensure these concepts are reliable rather than random noise, the current standard is to bootstrap the process by testing across 10,000 different seeds, which helps identify the most stable features, rank the features using a monosemanticity score, select features with high score (e.g., 80%). This helps ensuring the entries in the dictionary correspond to single, clear ideas with high correlation.

2. References

   1. XAI:

      • Alain, G., & Bengio, Y. (2017). Understanding intermediate layers using linear classifier probes. International Conference on Learning Representations (ICLR) Workshop.

      • Cunningham et al. (2023). Sparse autoencoders find highly interpretable features. arXiv:2309.08600.

      • Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Tetlock, A. (2018). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). International Conference on Machine Learning, 2668–2677.

      • Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. (2020). Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning (pp. 5338–5348).

      • Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30.

      • Belle, V., & Papantonis, I. (2021). Principles and practice of explainable machine learning. Frontiers in Big Data, 4, 688969.

      • Oikarinen, T. P., Das, S., Nguyen, L. M., & Weng, T.-W. (2023). Label-free concept bottleneck models. In International Conference on Learning Representations (ICLR).

      • Rao, V. (2025). The limits of AI explainability. arXiv:2504.20676.

      • Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I trust you?”: Explaining the predictions of any classifier. Proc. KDD’16, 1135–1144.

      • Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. International Conference on Machine Learning, 3319–3328. arXiv:1703.01365

   2. MI

      • Conmy, A., Mavor-Parker, A., Lynch, A., Heimersheim, S., & Garriga-Alonso, A. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS 36. arXiv:2304.14997.

      • Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., & Olah, C. (2022). Toy Models of Superposition. Anthropic Transformer Circuits Thread.

      • Nanda, N., & Meyer, B. (2022). TransformerLens: A Library for Mechanistic Interpretability of Generative Language Models.

   3. Surveys & Tools

      • Atf, Z. & Lewis, P. (2025). Is trust correlated with explainability in AI? A meta-analysis. arXiv:2504.12529.

      • De Silva et al. (2025). A multi-layered research framework for human-centered AI. arXiv:2504.13926.

      • Long et al. (2025). Explainable AI: The latest advancements and new trends. arXiv:2505.07005.

---