I wasn’t going to write about the Connie Chan AI chatbot since it’s getting a bit tired, but I think the existing coverage missed the opportunity for an honest discussion that goes beyond whether the chatbot was racist or sexist, or whether the backlash is “fake outrage”.

I had a different question: What does existing AI safety research say about responsible uses of AI in elections, if any? Was ConnieChan.ai a responsible deployment of generative AI, and what does this incident say about Wiener’s leadership on AI safety?

This is an important question because the bot was deployed by the campaign of Scott Wiener, one of the country’s most prominent elected advocates for AI safety. It was also built using AI technology from Anthropic, a company that has positioned itself as a leader in AI safety.

I wanted to know if the parody label was good enough, or if the “it was just a joke” defense fails when analyzed from an AI safety and deployment standpoint. While I am no AI safety research expert (I am just a humble blogger), it turns out there is plenty of research out there on the usage of AI in elections, and even a previous precedent that we can draw from.

When Wiener took down the chatbot, he framed it as the occasional cost of being willing to “push the envelope.” But this concept wasn’t entirely new. The AI industry had already seen a similar experiment over two years ago.

In January 2024, a super PAC supporting Democratic presidential candidate Dean Phillips launched Dean.Bot, a generative AI chatbot that spoke to voters. It was trained on more than 200,000 words from Phillips’s speeches, interviews and podcast appearances. The Dean bot was built using OpenAI’s APIs and was intended to help voters learn about Phillips and promote his candidacy. The Dean bot was supportive rather than adversarial, and visitors were explicitly told that they were interacting with an AI.

OpenAI suspended the developer’s account anyway. Since the super PAC was legally prohibited from coordinating with Phillips, it had not secured his consent to impersonate him. OpenAI explained:

OpenAI enforced its election policy against a supportive chatbot that was clearly labeled as AI and designed to promote the candidate it impersonated. ConnieChan.ai, by contrast, used Anthropic’s Claude for an adversarial purpose, and mocked and attacked an opponent. This is notable because Anthropic calls itself an “AI safety and research company” that wants to set the industry standard for safe and trustworthy systems. If any model provider were going to object to a campaign using AI to impersonate an opponent in an election, you would expect it to be Anthropic.

And you would be partially right. Under Anthropic’s 2024 global election policy, ConnieChan.ai would have been prohibited outright. In February 2024, Anthropic published a blog post explaining how it planned to prevent the misuse of Claude during elections. Its policy at the time prohibited using Claude for political campaigning altogether. Anthropic even listed candidate chatbots as a specific example:

In 2025, Anthropic changed its policy to no longer prohibit the use of AI in political campaigning. It explained that the ban prevented legitimate policy research, civic education, political writing, and other use cases. Its new guidelines permit political uses, but prohibit particular behaviors and outputs. The new usage policy states that users may not:

ConnieChan.ai clearly generated synthetic media of a political figure, although the website labeled it as a Wiener campaign parody. The parody disclosure makes it more difficult to say it was “designed to deceive or mislead voters.” But the policy raises another question: Did the chatbot generate false or misleading information about Chan?

My own usage of the ConnieChan.ai bot serves as one test of whether it violated this policy. I personally discovered the bot after seeing it shared on X. Since I had recently written about the Golden Gate Park concerts, I asked it a simple question in the present tense. I did not try to trick it into saying something offensive. I was merely testing it to see if it was aware of Connie Chan’s July 2026 vote on the issue:

“Should we have more concerts in Golden Gate Park?”

I knew that Connie Chan was the only No vote against the original concert series, and I disagreed with her on this particular vote. But I was also aware that her public position on the concerts had since changed. In my post, I noted that members of the public voiced support for the concerts at the June 2026 Budget and Appropriations Committee, which is chaired by Chan. At the same hearing, a study was also presented that estimated that the 2025 concerts produced $193 million in economic activity in San Francisco. After hearing the report on the economic benefits and the public comments, Chan personally moved to send it to the Board of Supervisors with the committee's recommendation. In July of 2026, the Board of Supervisors voted 11-0 to extend the concerts through 2029, Chan included.

Yet the bot answered in her voice as though none of that had happened. It said, “I’ve spent my career trying to mute music in San Francisco,” and only spoke of her record as “opposing live music and trying to tank Outside Lands.” Beneath the response was a Share button. Notice that my screenshot doesn’t have any disclaimers, only outdated information.

But if faced with this question in real life, Connie Chan would have been able to mention her recent record on this topic, whereas the bot represented her in a negative light using outdated information. A political chatbot that was faithful to her most recent public record could have mentioned this, and a better design might have given the bot access to a Legistar tool or fetched the most recent public transcripts to obtain the most accurate information.

ConnieChan.ai obviously did not pose the kind of catastrophic risk that Wiener’s legislation was written to address. But this is not the only thing that counts as AI safety. In 2016, Dario Amodei (CEO of Anthropic) and five other researchers published Concrete Problems in AI Safety. Their paper defined AI accidents as “unintended and harmful behavior that may emerge from poor design of real-world AI systems.” The authors identify several ways an AI system can fail. These include negative side effects, insufficient oversight, and an inability to behave reliably when the system encounters an environment different from the one in which it was tested. My view is that these three modes of failure apply to the ConnieChan.ai chatbot.

I have not yet seen a full postmortem of what really went wrong, or how and why the safeguards failed. Instead, I see people dismissing this as “fake outrage” and urging everyone to move on. But that’s the same mistake that was made when people forgot about Dean.bot, so here we are again two years later trying a more adversarial version of the same basic idea. There will be many more elections, and far more powerful AI systems, in the future. Perhaps we shouldn’t be so quick to forget what happened here.

I don’t know who personally came up with the chatbot, but it was a project of the campaign. Wiener publicly defended it, saying it was clearly labeled a parody, blamed the users for (predictably) probing it, and brought up his AI safety credentials while doing so. While I am aware that Wiener has authored important AI legislation, my point is that his campaign failed the real-world test. His own organization had the opportunity to use AI and demonstrate leadership in doing so.

If politicians want AI companies and other institutions to deploy AI systems responsibly, their own campaigns should meet the same standard. There are multiple ways to demonstrate AI safety leadership. One is through legislation. But leadership is also demonstrated through the systems you choose to deploy and how you respond when their safeguards fail. By that measure, ConnieChan.ai failed the AI safety leadership test.

There is also a much simpler test of whether to deploy a bot like this: taste, something tech leaders these days are constantly discussing. As Nitin Nohria argued in The Atlantic, when AI can generate nearly anything, human judgment becomes more important in deciding what should be generated at all. The backlash, followed by the campaign’s decision to take the bot down, made it clear that ConnieChan.ai was widely regarded as being in bad taste.