AI/ML SaMD

Bias and Generalizability: How FDA Evaluates AI/ML Training Data Representativeness in Medical Devices

By Andre D. Butler, medical device regulatory consultant, CQA, CCRP  ·  reviewed November 2026  ·  ← All Insights

Bias and Generalizability: How FDA Evaluates AI/ML Training Data Representativeness in Medical Devices

Photo by Brett Jordan on Unsplash

Why Your Training Dataset Is a Regulatory Document

When FDA reviewers open a submission for an AI/ML-based Software as a Medical Device (SaMD), they are not just evaluating your algorithm's performance metrics. They are auditing the population your model learned from -- and asking a pointed question: does this data actually represent the patients who will be affected by this device in the real world?

For founders and regulatory teams building AI-driven diagnostics, clinical decision support tools, or automated image analysis software, training data representativeness is not a data science afterthought. It is a core regulatory concern that can make or break a 510(k), De Novo, or PMA submission.

The Regulatory Foundation: What FDA Has Actually Said

FDA's expectations for AI/ML training data are grounded in several overlapping frameworks. The agency's 2021 action plan for AI/ML-based SaMD, the 2023 draft guidance on marketing submission recommendations for AI/ML-enabled devices, and the foundational principles established under 21 CFR Part 820 (Quality System Regulation, now harmonized with ISO 13485 under the 2024 Quality Management System Regulation) all converge on the same requirement: your data governance practices must be rigorous, documented, and defensible.

The 2023 draft guidance explicitly calls out the need for sponsors to describe their training, tuning, and test datasets in detail -- including the demographic composition of the patient population, the clinical sites contributing data, the imaging equipment or sensor types used, and the prevalence of disease and comorbid conditions represented. This is not boilerplate. FDA expects manufacturers to demonstrate that the intended use population is reflected in the data, not assumed into it.

Under 21 CFR 820.30 (Design Controls), now mirrored in 21 CFR Part 820 as updated, design validation must confirm that devices perform as intended for the target patient population. For AI/ML devices, that validation starts with the training corpus.

What FDA Means by 'Bias' in This Context

FDA uses the term bias in a technical, not colloquial, sense. Bias in AI/ML review refers to systematic errors introduced when the training data fails to represent the diversity of the clinical environment where the device will operate. This manifests in several forms that reviewers are specifically trained to identify:

  • Demographic underrepresentation: Models trained predominantly on data from one racial, ethnic, age, or sex group frequently underperform on underrepresented groups. FDA has flagged this explicitly in the context of dermatology AI, cardiac monitoring, and sepsis prediction tools.
  • Site and equipment bias: A model trained exclusively on data from a single academic medical center or one scanner manufacturer may not generalize to community hospitals or devices from other vendors.
  • Label bias: If ground truth annotations were generated by a narrow group of clinicians using non-standardized protocols, the model learns their systematic errors, not clinical reality.
  • Temporal bias: Data collected before significant shifts in clinical practice, disease prevalence, or patient demographics may render a model obsolete at the time of submission.

The Generalizability Standard FDA Applies in Practice

Generalizability is the affirmative obligation that mirrors the bias concern. FDA wants to see that you have affirmatively tested -- not just assumed -- that your model performs consistently across the subgroups, care settings, and device configurations present in your intended use population.

In practice, this means your submission should include stratified performance analyses across sex, age, race and ethnicity, disease severity, and any other clinically relevant subgroup. If your intended use spans multiple imaging platforms or laboratory instruments, you need performance data from each. If subgroup sample sizes are insufficient for independent statistical analysis, FDA expects a candid discussion of that limitation and a mitigation plan -- often in the form of a post-market data collection protocol or a predetermined change control plan (PCCP) under the 2024 PCCP guidance.

FDA has also signaled through warning letters and 510(k) not-substantially-equivalent decisions that simply reporting overall AUC or sensitivity without subgroup breakdowns is insufficient for AI/ML submissions. The agency's real-world performance monitoring expectations, articulated in the AI/ML action plan, extend this scrutiny into the post-market phase.

Practical Steps Before You File

Regulatory teams preparing AI/ML submissions should treat the following as non-negotiable pre-submission tasks:

  • Conduct a formal data representativeness analysis that maps your training dataset demographics against census data and clinical epidemiology for your intended use population.
  • Document your data curation and preprocessing decisions as part of your design history file under 21 CFR Part 820 -- including exclusion criteria and how they may introduce selection bias.
  • Define and pre-specify your subgroup analyses before you run your validation, so FDA cannot characterize them as post-hoc.
  • Engage FDA early through a Pre-Submission (Q-Sub) meeting to align on acceptable subgroup sample sizes and performance thresholds before you lock your test dataset.
  • Draft a PCCP that addresses distribution shift -- the scenario where real-world data drifts from your training distribution over time.

The Competitive Advantage of Getting This Right

Manufacturers who invest in rigorous training data governance do more than satisfy FDA. They build devices that actually work across diverse patient populations, reduce post-market surveillance burden, and position themselves for faster iterative updates under a well-constructed PCCP. In a regulatory environment where FDA is actively developing further AI/ML oversight frameworks, companies with mature data practices will adapt faster and face fewer surprises at review.

If your team is preparing an AI/ML SaMD submission and wants to pressure-test your training data strategy against current FDA expectations, the regulatory specialists at ADB Consulting and CRO Inc. are ready to help. Book a free discovery call at adbccro.com to discuss your device, your dataset, and the clearest path to a successful submission.

For related guidance, see our Software as a Medical Device regulatory support.

Prepared by ADB's Regulatory Team

Reviewed by Andre D. Butler, medical device regulatory consultant, CQA, CCRP

ADB's regulatory team brings 20+ years of hands-on FDA regulatory experience guiding medical device companies through 510(k), PMA, De Novo, AI/ML SaMD, and FDA 483 response engagements, with specialization in Section 524B cybersecurity compliance and ISO 13485 quality management systems across cardiovascular, orthopedic, diagnostic, and software-as-a-medical-device categories.

Ready to Navigate the FDA Process with Confidence?

Book a free 30-minute discovery call with ADB's regulatory team to discuss your specific device and situation.

Book a Free Pathway Call

Or call directly: (888) 450-8607

Explore our flat-fee FDA services →