Human-in-the-Loop Requirements for AI/ML Diagnostic Devices: What FDA Expects and How to Comply

Why Human-in-the-Loop Is No Longer Optional for AI/ML Diagnostic Devices

If your company is developing an AI/ML-enabled diagnostic device, the question is no longer whether FDA expects human oversight built into your system. The question is how robust, documented, and auditable that oversight needs to be -- and whether your current design controls and labeling can prove it.

Across 510(k) submissions, De Novo requests, and Pre-Submissions for AI/ML Software as a Medical Device (SaMD), FDA reviewers are scrutinizing human-in-the-loop (HITL) architecture with increasing precision. Founders and regulatory teams who treat HITL as a checkbox rather than a design philosophy are generating deficiencies, receiving Additional Information requests, and in some cases, watching submissions stall for months.

This post breaks down exactly what FDA expects, where the requirements come from, and how to structure your submission strategy accordingly.

The Regulatory Foundation: Where HITL Requirements Come From

There is no single regulation titled 'human-in-the-loop.' Instead, the requirement is woven across several overlapping frameworks that you need to read together.

• FDA's Artificial Intelligence and Machine Learning-Based Software as a Medical Device Action Plan (January 2021) -- This foundational document introduced the concept of a Predetermined Change Control Plan (PCCP) and emphasized that AI/ML SaMD must incorporate transparency and human oversight mechanisms as core design elements, not afterthoughts.
• Marketing Submission Recommendations for a Predetermined Change Control Plan for AI/ML-Enabled Device Software Functions (Draft Guidance, April 2023) -- This guidance explicitly links the credibility of a PCCP to the adequacy of human oversight protocols. If your device learns or adapts post-market, FDA wants to know how a qualified human stays in the loop during that evolution.
• 21 CFR Part 820 (Quality System Regulation) and the updated Quality Management System Regulation (QMSR), effective February 2026 -- Under the QMSR's alignment with ISO 13485:2016, design controls must demonstrate that risk management accounts for automation bias -- the clinically documented tendency of users to over-rely on algorithmic outputs. Your DFMEA and usability engineering files need to address this explicitly.
• IEC 62304 and FDA's Software as a Medical Device guidance (2017) -- SaMD classification drives the level of rigor required. A Class II diagnostic aid with a 'treat or diagnose' intended use under the SaMD framework carries significant obligations around user interface design and decision support transparency.

What 'Human-in-the-Loop' Actually Means in Practice

FDA does not require that a physician manually approve every algorithmic output. What the agency does require is that your device is designed so that a qualified human can meaningfully review, override, and take responsibility for any clinical decision the AI informs.

That distinction matters enormously. A radiology AI that flags a suspicious lesion and presents confidence intervals, supporting images, and a clear pathway to override the flag is designed for meaningful human oversight. A system that outputs a single binary result with no transparency into model reasoning and no documented override workflow is not -- regardless of how accurate the algorithm is on your validation dataset.

In practical terms, your HITL architecture should address the following:

• Display design and output transparency: Does your user interface present AI outputs in a way that supports informed clinical judgment rather than passive acceptance? Usability studies under FDA's Human Factors guidance (2016) should directly test this.
• Override mechanisms: Is there a documented, accessible pathway for the clinician to disagree with or disregard the AI recommendation? Is that action logged?
• Intended use and indications for use language: Your labeling must accurately characterize the AI as a decision support tool, not an autonomous diagnostic authority, unless your clinical evidence and regulatory pathway explicitly support autonomous function.
• Training and IFU requirements: FDA expects that operators understand the AI's limitations. Instructions for Use must define the intended user, the required clinical context, and the boundaries of the algorithm's validated performance.

Common Submission Deficiencies FDA Is Citing Right Now

Based on patterns in recent 510(k) deficiency letters and FDA feedback from Pre-Submission meetings, here are the gaps we see most often in AI/ML diagnostic submissions:

• Usability testing that evaluates task completion but does not assess automation bias or appropriate reliance behaviors
• PCCP documents that describe algorithmic retraining protocols without specifying what human review gates exist before updated models are deployed
• Indications for use language that overstates the autonomy of the AI, creating a mismatch with the HITL design the sponsor claims elsewhere in the submission
• Risk management files under ISO 14971 that identify automation bias as a hazard but do not trace mitigations back to specific design features or labeling controls

Strategic Recommendations for Founders and Regulatory Teams

If you are in early design, make HITL architecture a design input -- not a post-hoc compliance fix. The cost of retrofitting a user interface to satisfy FDA's human factors expectations after your verification and validation is complete is significant and avoidable.

If you are preparing a submission, request a Pre-Submission (Q-Sub) meeting with FDA specifically to align on your HITL approach before you invest in pivotal usability studies. FDA has been responsive to well-structured Q-Sub requests on AI/ML SaMD, and the feedback you receive will be binding on your reviewer.

If you have already received deficiencies related to human oversight or automation bias, do not respond with narrative assurances. Respond with data -- updated usability study results, revised risk management documentation with explicit HITL mitigations, and labeling edits that bring your indications for use into alignment with your design.

The Bottom Line

Human-in-the-loop is not a feature. It is a regulatory expectation embedded across FDA's AI/ML guidance framework, your design control obligations under 21 CFR Part 820 and the QMSR, and the risk management principles of ISO 14971. Companies that design for it from the beginning move faster through review, generate fewer deficiencies, and build products that clinicians actually trust.

At ADB Consulting and CRO Inc., Andre Butler works directly with medical device startups and established manufacturers to build AI/ML regulatory strategies that hold up under FDA scrutiny -- from Pre-Sub strategy through 510(k) submission and beyond.

Ready to pressure-test your AI/ML regulatory strategy? Book a free discovery call with Andre Butler at adbccro.com and get actionable guidance in your first conversation.

For related guidance, see our Software as a Medical Device regulatory support.