AI/ML Medical Device Submissions: What FDA Expects for Explainability and Interpretability

The Black Box Problem Is Now a Regulatory Problem

If your AI/ML-based medical device cannot explain how it arrives at a clinical decision, FDA will ask -- and your submission will stall. This is no longer a theoretical concern for future guidance. It is an active expectation showing up in pre-submission feedback, De Novo requests, and PMA review cycles right now.

Regulatory teams at startups and mid-sized device companies often underestimate how far FDA has moved on this issue. The agency is not asking for a general description of your algorithm. It wants structured, evidence-based documentation that demonstrates your model is transparent, validated across clinically relevant subgroups, and safe to deploy in a real-world clinical environment. If you are building an AI/ML Software as a Medical Device (SaMD), explainability is not a feature -- it is a submission requirement.

The Regulatory Foundation You Need to Know

FDA has built its AI/ML framework across several interconnected documents. Understanding how they work together is essential before you write a single line of your submission.

• FDA's Artificial Intelligence and Machine Learning in Software as a Medical Device Action Plan (January 2021): This document signaled FDA's commitment to a total product lifecycle approach, emphasizing that AI/ML models must be monitored and controlled even after market authorization.
• Predetermined Change Control Plan (PCCP) Guidance (December 2023): This guidance, developed in alignment with Section 3308 of the FDORA legislation, requires manufacturers to prospectively define how their AI/ML model may change and what performance boundaries govern those changes. Explainability feeds directly into how you define those boundaries.
• FDA's Discussion Paper: 'Transparency for Machine Learning-Enabled Medical Devices' (2021): While not a binding guidance, this paper introduced the concept of intended users needing to understand what the model outputs mean, what the model's limitations are, and under what conditions outputs should not be trusted.
• 21 CFR Part 820 (Quality System Regulation) and ISO 13485: Design controls under 21 CFR 820.30 require documented design inputs and outputs that address intended use. For AI/ML, this includes documenting the model's decision logic at a level sufficient to support design verification and validation.
• 21 CFR Part 880 and device-specific regulations: Depending on your device classification, predicate selection for 510(k) or De Novo pathways will shape how much interpretability evidence FDA expects in the substantial equivalence or risk-benefit analysis.

What Explainability Actually Means in a Submission Context

Explainability and interpretability are related but distinct. Interpretability refers to the degree to which a human can understand the internal mechanics of a model. Explainability refers to the degree to which the model's outputs can be communicated meaningfully to clinicians or end users, even if the underlying mechanics are opaque.

For regulatory purposes, FDA is primarily concerned with functional explainability -- can the intended user understand what the output means, when to trust it, and when to override it? This is especially critical for high-risk SaMD classifications (SaMD levels III and IV under the IMDRF framework) where AI outputs directly inform or drive clinical decisions.

Practically, your submission should address the following:

• Output transparency: What does the model output, and how is it displayed to the clinical user? Confidence scores, probability ranges, and uncertainty estimates must be documented and clinically validated.
• Feature attribution: For models where it is technically feasible, FDA increasingly expects some form of feature importance documentation -- whether through SHAP values, LIME, attention maps, or other post-hoc explanation methods. This is especially relevant for imaging AI and clinical decision support tools.
• Subgroup performance analysis: Your validation dataset must be interrogated across clinically relevant subgroups -- age, sex, race, comorbidity burden, imaging equipment type, and site of care. Unexplained performance disparities across subgroups are a direct explainability and safety concern.
• Failure mode documentation: FDA expects a structured analysis of conditions under which your model underperforms or produces unreliable outputs. This maps directly to your risk management file under ISO 14971.
• Clinician labeling and instructions for use (IFU): The IFU must explain the model's intended use, known limitations, and what actions a clinician should take when output confidence is low or when the device flags an edge case.

Where Submissions Break Down

The most common failure point is a mismatch between the technical depth of the algorithm documentation and the clinical framing of the intended use. Engineering teams document the model thoroughly for internal purposes but fail to translate that documentation into clinically meaningful terms that FDA reviewers -- and eventually clinicians -- can act on.

A second common gap is the absence of a prospective monitoring plan. FDA's PCCP guidance and the agency's broader lifecycle approach expect you to define, before authorization, how you will detect model drift, performance degradation, and real-world distribution shift. If you cannot explain how your model will be monitored after deployment, you cannot adequately explain how it will remain safe.

Strategic Recommendations Before You Submit

If you are preparing a 510(k), De Novo, or PMA submission for an AI/ML device, take these steps before you finalize your technical file:

• Request a Pre-Submission (Q-Sub) meeting with FDA to align on explainability expectations specific to your device classification and intended use.
• Engage your biostatistics team early to design a validation study with sufficient subgroup power -- retrofitting subgroup analysis after the fact rarely satisfies FDA review questions.
• Map your explainability documentation directly to your risk management file so FDA can see a clear line between identified risks, mitigation controls, and the information provided to end users.
• Review your IFU against FDA's draft guidance on clinical decision support software to ensure your labeling accurately characterizes the role of AI output in clinical workflow.

The Bottom Line

FDA is not asking device makers to abandon complex, high-performing models in favor of simple linear algorithms. The agency is asking for rigorous documentation that demonstrates you understand your model deeply enough to stand behind its clinical use. Explainability is the evidentiary bridge between your validation data and the clinician's trust in your device. Build that bridge before your submission -- not during the deficiency response cycle.

ADB Consulting and CRO Inc. works with AI/ML medical device companies at every stage of the regulatory pathway, from pre-submission strategy through PMA approval. If your team is navigating explainability requirements and needs a clear plan, we can help you build it.

Book a free discovery call with Andre Butler at adbccro.com and get a direct assessment of where your AI/ML submission stands and what it needs to succeed.

If this applies to your program, our AI/ML SaMD regulatory strategy guide walks through the process in detail.