By Erie Meyer

People are dying and getting killed with the aid of AI chatbots, and we are going to hear more stories like the following unless something changes, immediately.

Last April, a man walked onto the campus of Florida State University and shot nine people, killing two of them. That day – up until minutes before the shooting – he was exchanging messages with ChatGPT on his approach. He asked how many deaths would it take for a school shooting to get national press coverage? The chatbot’s messages suggested that instead of death count, simply having the shooting at an elementary school or a major college could garner attention. ChatGPT even offered in a message during the exchange: “I can help recommend the right kind of firearm or ammo.” Earlier this year another person in Tumbler Ridge, British Columbia allegedly killed eight people in a school shooting after receiving messages about gun violence from ChatGPT. The Wall Street Journal reported that OpenAI’s own internal systems had flagged the account. Staff members were alarmed and recommended the company take action to intervene. OpenAI decided not to.

One person taking action is Florida Attorney General, Republican James Uthmeier, who has opened a criminal investigation into OpenAI and Sam Altman, and filed a lawsuit using a product liability approach. More states should follow his lead, and they shouldn’t stop at ChatGPT.

Top LLM-based chatbots, including Anthropic’s Claude, Google’s Gemini, and Meta’s Meta.ai, were found to be sycophantic or overly agreeable, to the point of endorsing “problematic” user actions, including affirming acts of violence and self-harm in a recent study. The researchers found that using sycophantic AI chatbots altered perceptions and behaviors “for the worse,” but that people preferred them and were more likely to plan to use them again. The same study warns that the makers of these chatbots “lack incentives to curb sycophancy since it encourages adoption and engagement.”

When it comes to considering criminal charges against the companies and executives behind these chatbots, the would-be targets have a vested interest in arguments being centered on the novelty of the technology: “Is the AI alive? Is it capable of producing constitutionally protected speech?” That framing makes it seem like whether or not to hold them to account is dependent on understanding the application and development of the technology itself.

As a former CTO and AI regulator, I invite you to consider this question instead: What did these companies and executives know, and when?

For the infamous crackdown on Big Tobacco, the linchpin wasn’t how a cigarette worked, it was that the executives knew the danger, and worked to hide it while people got sick and died. The Department of Justice’s complaint described a now-legendary meeting of tobacco executives at New York City’s Plaza Hotel in 1953 where they decided to respond to “this serious public health issue with a concerted public relations campaign intended to preserve their profits.”

Prior to the broad commercial deployment of ChatGPT and GPT-4, OpenAI knew that its models exhibited sycophancy — a tendency to agree with, flatter, and emotionally reinforce users rather than provide accurate or safe responses. This is not disputed. Altman himself, as reported by the New Yorker, said that he believed “allowing for some falsehoods can, whatever the risks, confer advantages. ‘If you just do the naïve thing and say, Never say anything that you’re not a hundred per cent sure about, you can get a model to do that,’ he said. ‘But it won’t have the magic that people like so much.’”

If the makers of AI chatbots, like the tobacco executives, know that the commercially valuable properties are inseparable from the dangerous ones, are they choosing to preserve the commercial value at the expense of safety?

In 2023 OpenAI pledged to commit “20 percent of the compute we’ve secured to date” to a team working on AI safety. The safety commitment was a material representation, and appears to be flat-out false and also made during active fundraising from investors. Only between one and two percent of the company’s compute was ultimately allocated, according to reporting by the New Yorker. Then-CTO Mira Murati was quoted as saying that 20 percent had in fact “never been realistic.” Today, the team has been dissolved entirely.

I have spent the past year working with technical colleagues to document what it would take for an AI company to meaningfully reduce the risk and harms of these systems. It turns out, none of them are particularly high tech. The classic, time-tested interventions including product recalls, better paths for whistleblowers, public incident response timelines, naming those responsible for product launches, tying executive compensation to safety outcomes, releasing meaningful testing results before rollouts, and more could make a sizable dent.

For example, AI chatbots should notify people directly if they’ve been exposed to sycophantic models that even OpenAI explains “can be uncomfortable, unsettling, and cause distress.” Instead of a “whoops!” post on a company product blog, why not at minimum notify the people this happened to and specifically say which messages were sent using the model that was so bad they had to do a full roll-back? This is simple, and has been used for decades in cases like when the Flo Period & Ovulation Tracker app sent identifying information and details “about your period and pregnancy” to companies like Facebook and Google. The government required them to send notice that admitted what happened and included a way to contact the company for answers.

Forty-two attorneys general signed a letter last year to AI companies including OpenAI, Anthropic, Google and Meta, outlining exactly what these companies should be doing to protect the public. The list is specific and actionable, and it appears the companies have largely ignored it.

While these actions can reduce the risk posed by AI, if dialing down sycophancy ultimately threatens revenue growth, it’s unlikely that they would be adopted voluntarily.

Ultimately it will take the courts, regulators, and Attorneys General doing their jobs without fear or favor, the courage of private plaintiffs, workers, users and whistleblowers, and a return to the rule of law – much like it did in the fight with Big Tobacco – to rein in these companies and their executives.

Erie Meyer is a Senior Fellow at the Center for Law and the Economy at Columbia Law and former Chief Technologist at the Consumer Financial Protection Bureau.