AI in Healthcare: HIPAA, FDA Oversight, and Legal Considerations

Artificial intelligence is being deployed across the healthcare sector at a pace that has outrun the legal framework governing it. AI tools assist radiologists in detecting tumors in medical images, help physicians identify patients at risk of clinical deterioration, power chatbots that triage patient symptoms, support pharmacists in reviewing medication orders for dangerous interactions, and automate administrative functions from prior authorization to revenue cycle management. Each of these applications raises a distinct set of legal questions, and the organizations deploying them — from large health systems to small clinical practices to healthcare technology startups — need to understand the legal terrain before they deploy AI in patient care.

HIPAA and AI: The Data Problem

AI systems in healthcare require training data, and the most valuable training data for clinical AI is patient health information. HIPAA governs how that data can be used. Protected health information can generally be used or disclosed for treatment, payment, or healthcare operations without patient authorization, and some AI applications — those that directly support clinical operations — may be permissible under these exceptions. But using patient data to train AI systems that will be deployed broadly, sold to third parties, or used for purposes beyond the direct care of the patients whose data was used raises more complex questions.

HIPAA’s research exception allows covered entities to use PHI for research purposes, but only with specific patient authorization, an IRB or privacy board waiver of authorization, or use of de-identified data that no longer qualifies as PHI. De-identification under HIPAA requires either the removal of all 18 specified identifiers or a formal statistical determination by a qualified expert that the risk of re-identification is very small. The challenge is that highly de-identified data is often less useful for training AI models that need to recognize subtle patterns in patient populations. Healthcare AI developers and the covered entities that partner with them must navigate the tension between data utility and HIPAA compliance carefully.

When a healthcare organization shares patient data with an AI vendor for the purpose of training or operating an AI system, that vendor is almost certainly a business associate under HIPAA. A business associate agreement must be in place before any PHI is shared. The BAA must define the purposes for which the vendor can use the PHI, and using patient data to train AI models that will be sold to other customers or used to benefit parties other than the covered entity is not a use that most standard BAAs contemplate. Healthcare organizations should review their vendor agreements carefully to ensure that the AI vendor’s use of patient data is properly authorized and documented.

FDA Regulation of AI Medical Devices

The Food and Drug Administration regulates AI software intended to be used for medical purposes as a medical device. The FDA’s regulatory authority over software as a medical device — called SaMD — has evolved significantly over the past decade as AI has become more prevalent in clinical applications. Not every software application used in healthcare is a regulated medical device: software that performs only administrative, financial, or general clerical functions is generally not subject to FDA oversight. But software that is intended to diagnose, cure, treat, or prevent a disease or condition, or that is intended to affect the structure or function of the body, is a medical device and is subject to FDA regulation.

AI systems that analyze medical images to detect cancer, algorithms that predict patient deterioration, systems that recommend treatment options, and tools that assist in dosing medications all potentially qualify as medical devices. The FDA evaluates AI medical devices under its pre-market review framework, which requires manufacturers to demonstrate reasonable assurance of safety and effectiveness before bringing a product to market. Most lower-risk AI medical devices are subject to 510(k) clearance, which requires demonstrating substantial equivalence to a legally marketed predicate device. Higher-risk AI devices may require pre-market approval, which involves a more rigorous clinical evidence standard.

A particular challenge for the FDA’s regulatory framework is that AI systems can change their behavior over time as they are exposed to new data. A traditional medical device — a surgical instrument, an imaging machine — performs the same function consistently and does not change based on new inputs. An AI system designed to continuously learn from new patient data may perform differently after deployment than it did when it was initially approved. The FDA has issued guidance on this issue, distinguishing between locked AI systems (which do not change their behavior after deployment) and adaptive AI systems (which can change), and has proposed a predetermined change control plan approach that would allow manufacturers to make certain types of updates to their AI systems without re-submitting for full regulatory review.

AI in Administrative Healthcare Operations

Not all healthcare AI applications are medical devices subject to FDA oversight. AI used for administrative purposes — scheduling, prior authorization processing, revenue cycle management, clinical documentation assistance, coding support, and similar functions — is generally not a regulated medical device. But this category of AI is still subject to HIPAA when it processes protected health information, and it may be subject to other legal requirements depending on the specific function.

AI used in prior authorization decision-making deserves particular attention. The use of AI to make or support coverage determinations has drawn scrutiny from insurers, regulators, and plaintiffs’ attorneys. The Centers for Medicare and Medicaid Services has indicated that prior authorization determinations based on AI cannot be made without adequate individualized review, and several states have enacted legislation requiring that prior authorization decisions involving AI be subject to review by a licensed healthcare professional. Insurers and health plans deploying AI for coverage determinations should carefully evaluate compliance with these requirements.

Liability for AI-Assisted Clinical Decisions

When an AI system contributes to a clinical decision that results in patient harm, the question of who bears legal responsibility is genuinely complex. Under traditional medical malpractice law, healthcare providers have a duty to exercise the standard of care that a reasonably competent provider in the same specialty would exercise. Using AI in clinical decision-making does not eliminate this duty — it changes how it is fulfilled. A physician who relies on an AI diagnostic recommendation without exercising independent clinical judgment may not be meeting the standard of care if a reasonably competent physician would have recognized that the AI recommendation was inconsistent with the clinical picture.

At the same time, healthcare providers who fail to use available AI tools that could have identified a serious condition may face claims that they fell below the standard of care by not using the technology a reasonably competent provider would have used. As AI tools become more prevalent and their use becomes more standard in certain specialties, the standard of care itself may evolve to require their use in appropriate clinical contexts.

The AI vendor that developed the system may also face claims if the system was defective or if it was marketed for uses beyond its validated performance. Product liability, negligence, and breach of warranty claims against AI medical device manufacturers are an emerging area of litigation. Healthcare organizations that are deploying AI in clinical settings should understand the liability allocation provisions in their agreements with AI vendors and should consider whether their professional liability insurance covers adverse events in which AI played a role.

Algorithmic Bias in Healthcare AI

Healthcare AI trained on historical data can perpetuate and amplify existing disparities in healthcare access and quality. If the training data underrepresents certain patient populations, the AI may perform less accurately for those populations. If historical clinical practice reflected biased treatment decisions, an AI trained on that data may make similarly biased recommendations. Several high-profile studies have documented racial bias in healthcare AI systems, including an algorithm widely used to identify patients for high-risk care management programs that used healthcare costs as a proxy for health needs — resulting in systematic underidentification of Black patients who had equal or greater clinical needs but lower historical costs due to reduced access to care.

Healthcare organizations deploying AI in clinical or administrative settings have both ethical and legal obligations to evaluate their AI systems for bias and to implement safeguards that prevent discriminatory outcomes. Section 1557 of the Affordable Care Act prohibits discrimination in healthcare on the basis of race, color, national origin, sex, age, and disability, and regulatory guidance has indicated that this prohibition extends to AI-assisted decision-making. Healthcare organizations should ask AI vendors to provide data on their systems’ performance across demographic groups and should conduct their own evaluation of AI performance on their patient populations before deployment.