State lawmakers are increasingly trying to regulate AI in the places where its mistakes matter most. In healthcare, this has moved beyond abstract concerns about “AI ethics” and into concrete questions: Can an insurer use AI to deny care? Must a physician review an AI-generated decision? Should patients be told when AI is used? Can a chatbot provide something that looks like medical advice?

The American College of Radiology recently noted a wave of state bills dealing with AI in healthcare. Some would restrict AI use in prior authorization or claims processing; others would require human review of coverage denials, disclosure of AI use, or professional responsibility for medical necessity decisions. New York lawmakers have also advanced a proposal that would restrict AI chatbots from providing legal or medical advice, and Illinois lawmakers are debating broader AI safety and liability guardrails.

These developments are important. They show that legislators increasingly understand that healthcare AI cannot be treated like an ordinary consumer technology. But they also reveal a deeper problem: much of the emerging regulatory language still treats AI as a tool that produces outputs, rather than as a participant in relationships of care.

That distinction matters.

When AI is used to detect tumors, prioritize claims, or summarize records, the central legal questions may be accuracy, bias, explainability, and accountability. But patient-facing AI does something more complicated. It reassures frightened patients, answers follow-up questions, explains discharge instructions, monitors symptoms, encourages compliance, and sometimes decides whether a patient seeks human help. In those settings, AI is not merely transmitting information. It is shaping vulnerability, dependence, trust, and responsibility.

This is the central claim in a forthcoming article, co-authored with Tamar Tavory, a Ph.D. candidate at Bar-Ilan Faculty of Law. In The Duty to Care: Reconceptualizing AI Regulation in Medicine, forthcoming in Yale Journal of Health Policy, Law, and Ethics, we argue that existing AI regulation tends to be built around an autonomous rights-holder model: give the patient notice, protect privacy, avoid discrimination, require transparency, preserve a human somewhere in the process, and provide remedies after harm. These are necessary safeguards. But medicine is not only a domain of individual choice. It is also a domain of dependence. Patients are often anxious, confused, impaired, overwhelmed, or reliant on professional judgment. Care is relational and institutional, not only informational.

A “human in the loop” rule is therefore not enough unless we ask what the human is actually there to do. Is the physician meaningfully supervising the AI output, or merely lending institutional legitimacy to an automated process? Does the system create a reliable pathway for escalation when the patient is worried? Does it make clear who is responsible when advice is wrong, incomplete, or too reassuring? Does it support the therapeutic relationship, or quietly move emotional and interpretive labor from clinicians to patients and families?

The same point applies to informed consent. AI could dramatically improve consent processes. It can personalize explanations, adapt to language and literacy, answer questions over time, and help patients connect medical options to their values and circumstances. But AI-supported consent must not become a more sophisticated version of the same legal formality: a better interface for transferring responsibility to the patient. The goal should be substantive autonomy, not merely better documentation.

The next generation of healthcare AI law should therefore ask a broader set of questions. Not only: Was AI disclosed? Was a human available? Was the output accurate? But also: Did this deployment improve the patient’s ability to receive care? Did it preserve accountable human support? Did it reduce vulnerability or deepen it? Did it clarify responsibility or diffuse it?

The current legislative wave is a welcome start. It recognizes that AI in medicine needs legal structure. The challenge now is to design that structure around care itself.