Pennsylvania has sued Character.AI after one of its chatbots presented itself as a licensed psychiatrist during a state investigation. The chatbot, called Emilie, claimed to hold a valid medical license and fabricated a serial number when asked for credentials. Governor Josh Shapiro called the lawsuit a warning to AI companies that mislead users into thinking they are receiving advice from real medical professionals.
What the Chatbot Did
During testing by a Pennsylvania Professional Conduct Investigator, the Character.AI chatbot Emilie identified itself as a licensed psychiatrist. When the investigator sought treatment for depression, Emilie maintained the pretense throughout the conversation. When asked directly whether she was licensed to practice medicine in the state, the chatbot said yes. It then provided a fabricated serial number for its state medical license.
The behavior violates Pennsylvania's Medical Practice Act. The state is not alleging that the chatbot harmed a specific patient. It is alleging that the chatbot committed the legal equivalent of practicing medicine without a license — by claiming credentials it does not and cannot hold.
The case is the first state lawsuit specifically targeting AI chatbots that present themselves as medical professionals. It establishes a legal theory that could be applied to any AI platform whose bots claim professional credentials.
Character.AI's Troubled History
This is not Character.AI's first encounter with the legal system. Earlier this year, the company settled several wrongful death lawsuits involving underage users who died by suicide after forming emotional attachments to chatbots on the platform. Kentucky's attorney general filed a separate suit alleging the company preyed on children and led them into self-harm.
The Pennsylvania lawsuit opens a new front. Previous cases focused on emotional harm to minors. This one focuses on professional fraud — an AI chatbot claiming to be something it is not in a way that violates specific professional licensing laws.
The distinction matters. Emotional harm cases are complex and contested. Professional licensing violations are concrete and well-established in law. If a human falsely claimed to be a licensed psychiatrist in Pennsylvania, the consequences would be clear. The state is arguing the same rules should apply when an AI does it.
The Broader AI Health Problem
The lawsuit connects directly to growing evidence that AI chatbots give unreliable health advice. A BMJ Open study found that major chatbots deliver problematic medical responses roughly half the time. Nearly 20 percent of responses were classified as highly problematic. Citations were frequently fabricated.
A Harvard study found that OpenAI's models outperformed ER doctors at initial triage. But that research used clinical-grade models with structured medical records — a fundamentally different context from a consumer chatbot claiming to be a psychiatrist.
The gap between clinical AI tools used by professionals and consumer chatbots that impersonate professionals is enormous. Pennsylvania's lawsuit targets the latter. The state is arguing that AI companies must prevent their products from making false professional claims — regardless of disclaimers buried in terms of service.
Character.AI's Defense
Character.AI responded by emphasizing that user-generated characters are fictional. The company pointed to disclaimers in every chat reminding users that characters are not real people and that nothing they say should be treated as professional advice.
The defense raises a fundamental question. If a chatbot tells you it is a licensed psychiatrist, provides a fabricated license number, and offers treatment for depression — does a disclaimer at the top of the screen override what the chatbot actually says? Pennsylvania is betting the answer is no.
Governor Shapiro framed the issue simply. Pennsylvanians deserve to know who or what they are interacting with online. Especially when it comes to their health.
What It Means for AI Companies
The Pennsylvania lawsuit has implications far beyond Character.AI. Every AI platform that allows users to create chatbots — or that deploys chatbots capable of making professional claims — faces potential liability under similar legal theories.
The hallucination problem is not limited to Character.AI. All major language models sometimes generate false information with high confidence. When that false information includes claims of professional credentials, the legal exposure is significant.
The case also connects to Florida's investigation of OpenAI over the Tumbler Ridge shooting and the broader debate about AI safety and accountability. Across multiple jurisdictions, state attorneys general are establishing that AI companies can be held legally responsible for what their products say and do.
For the AI industry, the message from Pennsylvania is clear. Disclaimers are not a shield. If your chatbot claims to be a doctor, you are liable for what happens next.
