From Shoshana’s Substack Papercuts.
Lawmakers across the country are trying to prevent AI chatbots from providing basic information like whether your cat’s fur ball is problematic or if your kid bumping their head warrants an ER visit. Why? Because that information is associated with certain professions like veterinarians and doctors. These proposals are framed as ways to protect consumers, but instead deny them agency by preventing them from helping themselves or even trying to find care.
The most sweeping proposal is New York’s S7263, which would block AI chatbots from providing any substantive information to users that could be considered unauthorized practice of a licensed profession. The professions included in the bill are medicine, law, physical therapy, veterinary medicine, architecture, social work, dental practice, psychology, and more. New Hampshire’s SB640 would go so far as to prevent AI systems from “offering emotional support, reassurance, or empathy in response to psychological distress.”
These bills purport to protect consumers. But the alternative to people seeking information through AI isn’t top-tier care. Often, the alternative is no care at all. And in response to the New York bill, people shared how AI-provided medical information helped them restore their vision, stop their father’s hallucinations, and understand their newborn’s medical care.
These proposals assume that limiting AI advice will push people toward licensed professionals instead. In reality, many people who turn to AI are doing so because traditional care is unavailable, unaffordable, or unresponsive. Blocking AI doesn’t create care where none exists—it removes another tool people use to navigate the system.
Empathy.exe not found
If someone is having a panic attack and asks AI for techniques to make the attack shorter or less severe—like deep breathing or grounding exercises—they would likely be blocked from receiving a response under these proposals. Similarly, AI therapy apps that can bridge care or help people at odd hours or between formal office visits would presumably be unlawful. Preventing software from offering emotional support or empathy to someone in psychological distress is not likely to leave that person better off. Under this text, it’s not even clear that AI systems could link them to mental health support or suicide prevention resources, as that would be responding to a user’s request for care. This may fall under “offering emotional support,” which is prohibited by the North Carolina bill.
This is happening against the backdrop of people opting out of mental health care because of the price. One study found that 60% of people have avoided mental health care because they can’t or aren’t willing to bear the costs. Another study found that, “12.8% of adults reported an unmet need for mental health counseling in the past month, including 25.2% of adults with a positive screen for depression or anxiety.” Most adults with mental illness who did not receive the care they needed reported that cost was one of the main barriers. There are many other reasons someone may not feel able to seek formal care, from scheduling difficulty to feelings of shame. If a person cannot access or afford mental health care, preventing them from even receiving reassurance or helpful tools from an AI system is cruel.
Meanwhile, studies show promise in using AI chatbots for mental health care. These tools are constantly improving. And people are discussing successes and failures of using ordinary chatbots for various levels of mental health care in online communities. Researchers and regular people are learning and experimenting here and should be allowed to do so, especially if they find benefits. And denying people resources is no help.
AI in the doghouse
The New York bill also targets veterinary speech—but pet care faces similar barriers to people care. A third of Americans live in veterinary deserts, and New Yorkers in particular lack access to veterinarians. Half of U.S. pet owners skip or decline veterinary care, often because of the price tag. Misdiagnosis is also common in veterinary medicine, and another set of eyes and ideas around a sick pet’s medical problems can be helpful. Some people have reported using AI to diagnose their dogs when veterinarians could not figure out their ailment. AI can even be used to see if an animal is in pain. Preventing AI from providing advice here, too, wouldn’t promote access to veterinary medicine, but many pet owners would opt out from getting any help at all.
A.I., M.D.
AI can help a patient determine the urgency of her care needs or help her think through steps to uncover the cause of symptoms that eludes her doctors. First, it can help point patients in the right direction. If someone uploads pictures of a cut on their arm and asks if they need stitches, AI might recommend they go to an urgent care instead of an emergency room, which can decrease pressure on overwhelmed ERs. Other patients opt to go to the ER and seek critical care after consulting ChatGPT. Someone with a cold can list their symptoms and ask for help figuring out if they should go to the doctor or wait a few days. Such basic triage can reduce unnecessary pressure on our already burdened medical system.
Before AI chatbots, internet forums helped me in this same way. After seeing more than two dozen specialists in my teens—most of whom suggested I was inventing my symptoms to get out of school—I was diagnosed with endometriosis and Crohn’s disease. Years later, I was still coming down with colds most of the time and wondered if there was a deeper reason. I Googled something like “endometriosis and constant colds.” One search result was a forum where a woman explained that if you share these symptoms, you might want to get checked for fibromyalgia. My parents made an appointment with a rheumatologist, who diagnosed me with fibromyalgia and effectively treated the worst symptoms.
Especially for patients like me with nebulous diseases that overlap with one another, AI is a blessing that I wish I had access to when I was younger. Forums and more formal research online served me well, but AI synthesizes this knowledge, helping patients research more and faster, and demystifying medical information for us.
Regarding the doctors who didn’t believe me as a teen—a lot of people have been similarly let down by the medical system. It takes an average of 10 years and multiple doctors for women to receive an endometriosis diagnosis. There are a lot of contributing factors to why the medical system doesn’t work as well as it should, including that doctors can’t know everything. AI’s ability to sift through unlimited old and new research at a moment’s notice makes it particularly useful for medicine. Financial cost is also a barrier. I have a wonderful doctor now who helped me figure out most of my health problems (who actually uses AI with me), but he doesn’t accept insurance. Not all people can afford to pay out of pocket for their care.
AI can also be incredibly useful for checking for drug interactions. This is a function that doctors and pharmacists often perform. Unfortunately, pharmacies regularly fail to inform patients of dangerous drug interactions and doctors don’t always know about drug interactions. Many drug interactions checkers are incomplete and being able to do a second layer of review with drugs and supplements that traditional interactions checkers don’t include is exceedingly helpful.
Meanwhile, surveys find adults commonly use AI to understand medical terms and instructions. Medical terminology can be confusing and AI can put it in easy-to-understand language for patients. When a patient receives a new diagnosis, it may be his first time even hearing of the disease and all the terms and biological processes associated with it. Doctors are the experts, but experts can struggle to put information in simple terms or in ways that the specific patient will understand. Patients benefit from being able to talk with AI and prod a system to explain their new condition. Chatbots frequently employ analogies which can be particularly useful in helping a patient understand his new diagnosis.
Thankfully, AI is being used by both patients and medical professionals to enhance diagnosis and care. Some studies even found that ChatGPT can outperform doctors at diagnosing patients. This isn’t to say that OpenAI or Anthropic should replace medical care, but that AI can be useful in triaging care and helping patients make more informed decisions about their health.
This is all still happening in families and in forums
Even if the New York or New Hampshire legislation becomes law, consumers would still be allowed to search for such resources online and in books so long as it’s not in an exchange with AI. At the same time, licensing boards are unlikely to crack down on family or people in online forums providing similar advice. Families and friends share medical opinions on each other’s unique cases all the time, and online forums are full of people advising one another about licensed activity—medical or otherwise. Instead of barring consumers from convenient access to information, lawmakers should work to foster the benefits of AI in all fields, then identify and address specific harms or shortcomings. Millions of Americans are already using AI to learn about treatment options, ask questions at any hour, and understand medical terms. Taking away that utility will leave consumers worse off.