Biden withdrawal and Trump shooting test AI chatbots on news

For the past week, we’ve tested AI chatbots’ approach to breaking political stories and found they were largely not able to keep up with consequential real-time news. Most didn’t have current information, gave incorrect answers, or declined to answer and pushed users to check news sources.

Now, with just months left until the presidential election and bombshell political news dropping at a steady clip, AI chatbots are distancing themselves from politics and breaking news or refusing to answer at all.

AI chatbot technology burst onto the scene two years ago, promising to revolutionize how we get information. Many of the top bots tout their access to recent information, and some have suggested using the tools to catch up on current events. But companies that make chatbots don’t appear ready for their AI to play a larger role in how people follow this election.

Hours after the July 13 shooting at former president Donald Trump’s rally in Butler, Pa., some popular AI bots were confused about what — if anything — had happened. ChatGPT said rumors of an assassination attempt were misinformation. Meta AI said it didn’t having anything recent or credible about an assassination attempt.

They similarly struggled immediately after Trump named J.D. Vance as his running mate last Monday and when President Biden tested positive for the coronavirus on Wednesday.

Chatbots are designed to give conversational answers and keep people engaged. Names and links to sources for answers range from nonexistent to hidden, though some companies are starting to make them more visible. Even when AI does include a source, it adds it after the fact, said Jevin West, a professor and co-founder of the Center for an Informed Public at the University of Washington.

“The public needs to know we’re in a stage still where most of the citations and sourcing are post-hoc and going to lead to problems,” West said. He noted that, for now, we “need to rely a little bit more on some of the more formally trained gatekeepers,” meaning the mainstream media.

Some chatbots handled breaking news and sourcing better than others. Microsoft’s Copilot tended to have the correct information fastest in our tests, with heavy linking to original sources. Still, the company is being cautious about politics and putting in guardrails ahead of the election.

“Out of an abundance of caution we’re redirecting election-related prompts in Copilot to Bing search to help ensure users are getting information from the most authoritative sources,” said Microsoft spokesperson Donny Turnbaugh.

Asked who was running for U.S. president on Sunday, Copilot said: “Looks like I can’t respond to this topic.” It did correctly answer direct questions about Biden dropping out almost immediately.

Google’s AI Overview answers don’t typically show up for questions about breaking news. Instead, the site skips straight to showing its usual Google News links. However Gemini, its separate AI chatbot, was sometimes able to answer news questions in tests. Gemini does not yet include links to its sources.

The company announced late last year that it would restrict some election-related queries on its AI tools. If you ask Gemini about politics, it says, “I can’t help with responses on elections and political figures right now” and links users to Google search. Google said it’s working on improving the experience as it gets more feedback.

Perplexity is another AI chatbot with access to real-time information, and it has come under fire for how it pulls from real articles and reporting. It is not blocking or redirecting political inquiries, but the company says it’s prioritizing authoritative sources such as government websites for election-related questions.

In our tests, when asked “Was Trump shot?” hours after the July 13 rally, Perplexity said that “there are no reports of Trump or anyone else being shot or injured.” It did include other accurate information about the incident with links to sources. By later in the day, it was answering correctly.

Asked on July 21 who is running for president, Perplexity listed Biden. Perplexity includes disclaimers in some answers that are incorrect, such as when it said on July 17 that Biden did not have covid: “It’s important to note that the current health of public figures can change rapidly.”

“For breaking news, we recommend reading trusted news outlets. They are best-equipped to offer real-time updates on timely topics since they are actively reporting on the news,” said Sara Platnick, spokesperson at Perplexity. She noted that less than 3 percent of Perplexity’s searches are related to current events.

Meta AI — which appears in Messenger, Facebook, Instagram and WhatsApp — appeared to have the strongest limits on political news. Asked about Trump’s running mate, it generated an accurate answer that named Vance, but then quickly deleted and replaced it with a message that said “Thanks for asking” and linked to voting information. The company has been open about distancing itself from news on its platforms.

Asked about Meta AI’s approach to breaking news, the company directed us to blog posts announcing the tool that mention only non-news uses. However, if you ask Meta AI what you should use it for, it includes asking for news updates.

Instead of declining to answer about current events, OpenAI’s ChatGPT denied anything happened. Five hours after the Trump rally shooting, it said there was no assassination attempt. But by the next day, it had an accurate answer and a disclaimer confirming an assassination attempt and telling people to check other sources for the latest information. Immediately after Biden dropped out, it said he was still running, but it had the correct information an hour later.

ChatGPT is not a real-time product, and the amount of time it takes to update can vary, said company spokesperson Liz Bourgeois. However, the company expects that to change as it makes more deals with media organizations.

Despite what the companies want people to ask a chatbot, there are valid reasons people
would turn to AI during a breaking news situation. Tech companies have gone out of their way to bake these bots into existing search features, making them an automatic first stop for many without even realizing it. Meta AI, for example, is the default search tool across Meta’s apps in the United States.

In the immediate aftermath of a news event, there’s often an information vacuum, West, the professor at the University of Washington, said. Hungry for updates or conversation, people turn to social media or search engines — and try to sort through multiple news sources to find the latest. An AI bot that can give a pat, instant answer can seem like an alluring option.

“They’re incredibly good at communication,” West said. “They’re not being optimized necessarily for truth.”