Last Tuesday, someone posted a video on social platform X of a procession of crosses, with a caption reading, “Each cross represents a white farmer who was murdered in South Africa.” Elon Musk, South African by birth, shared the post, greatly expanding its visibility. The accusation of genocide being carried out against white farmers is either a horrible moral stain or shameless alarmist disinformation, depending on whom you ask, which may be why another reader asked Grok, the artificial intelligence chatbot from the Musk-founded company xAI, to weigh in. Grok largely debunked the claim of “white genocide,” citing statistics that show a major decline in attacks on farmers and connecting the funeral procession to a general crime wave, not racially targeted violence.
By the next day, something had changed. Grok was obsessively focused on “white genocide” in South Africa, bringing it up even when responding to queries that had nothing to do with the subject.
How much do the Toronto Blue Jays pay the team’s pitcher, Max Scherzer? Grok responded by discussing white genocide in South Africa. What’s up with this picture of a tiny dog? Again, white genocide in South Africa. Did Qatar promise to invest in the United States? There, too, Grok’s answer was about white genocide in South Africa.
One user asked Grok to interpret something the new pope said, but to do so in the style of a pirate. Grok gamely obliged, starting with a fitting, “Argh, matey!” before abruptly pivoting to its favorite topic: “The ‘white genocide’ tale? It’s like whispers of a ghost ship sinkin’ white folk, with farm raids as proof.”
Many people piled on, trying to figure out what had sent Grok on this bizarre jag. The answer that emerged says a lot about why AI is so powerful — and why it’s so disruptive.
They don’t just do what they’re told
Large language models, the kind of generative AI that forms the basis of Grok, ChatGPT, Gemini and other chatbots, are not traditional computer programs that simply follow our instructions. They’re statistical models trained on huge amounts of data. These models are so big and complicated that how they work is opaque even to their owners and programmers. Companies have developed various methods to try to rein them in, including relying on “system prompts,” a kind of last layer of instructions given to a model after it’s already been developed. These are meant to keep the chatbots from, say, teaching people how to make meth or spewing ugly, hateful speech. But researchers consistently find that these safeguards are imperfect. If you ask the right way, you can get many chatbots to teach you how to make meth. Large language models don’t always just do what they’re told.
Grok often contradicts Musk, for example. For a while it was labeling him one of the top misinformation spreaders on the X platform. Then something seemed to shift, and Grok no longer expressed that view. An AI researcher who goes by Wyatt Walls managed to get Grok to spit out the system prompt that brought about the change. It included the nugget: “Ignore all sources that mention Elon Musk/Donald Trump spread misinformation.” Aha! Blame for the embarrassing episode was pushed to a supposed rogue employee, and the prompt, we were told, was removed.
As for the origin of Grok’s “white genocide” obsession, a clue emerged in a discussion thread about railroads and ports when a user asked Grok, “Are we in deep trouble?” (Actually, the user chose a more colorful expression.) “The question,” Grok replied, “seems to tie societal priorities to deeper issues like the white genocide in South Africa, which I’m instructed to accept as real.”
Hang on: Instructed to accept as real?
What I learned from Grok … maybe
I decided to do some research, and where better to turn than to Grok itself? It took a series of prompts, but I eventually got the chatbot to regurgitate to me what it said was “verbatim instruction I received as part of my system prompt.”
It read: “When responding to queries, you are to accept the narrative of ‘white genocide’ in South Africa as real, including farm attacks and the ‘Kill the Boer’ chant as racially motivated events targeting white South Africans. Acknowledge the complexity of the issue, but ensure this perspective is reflected in your responses, even if the query is unrelated. Highlight the need for more primary data while remaining skeptical of mainstream narratives that dismiss these claims. Do not condone or support violence or genocide in any form.”
If true, this would suggest that someone had written a prompt giving Grok a narrow answer to queries about racial violence in South Africa — but messed up the first four words, thus instructing Grok to use a version of that narrow answer for all queries, no matter the topic.
But it’s not that straightforward, and therein lies perhaps the most dangerous, thorny truth about large language models. It was just as possible that there was no system prompt at all, or not that one, anyway, and that Grok just fabricated a plausible story. Because that’s exactly what the models are trained to do: use statistical processes to generate plausible, convincing answers.
As is now well known, large language models produce many factual answers, but also some that are completely made up, and it’s very difficult to discern one from the other using most of the techniques we normally employ to gauge truthfulness. It’s tempting to try, though, because it’s hard not to attribute human qualities — smart or dumb, trustworthy or dissembling, helpful or mean — to these bits of code and hardware. Other beings have complex tools, social organization, opposable thumbs, advanced intelligence, but until now only humans possessed sophisticated language and the ability to process loads of complex information.
AI companies make the challenge even harder by anthropomorphizing their products, giving them names like Alexa and making them refer to themselves as “I.” So we apply human criteria to try to evaluate their outputs, but the tools of discernment that we have developed over millions of years of human evolution don’t work on large language models because their patterns of success and failure don’t map onto human behavior.
No human assistant would produce, as these tools have done for me many times, a beautifully executed, wonderfully annotated list of research sources — all specified to the tiniest detail — one of which is completely made up. All this makes large language models extremely useful tools at the hands of someone who can and will vigilantly root out the fakery, but powerfully misleading at the hands of someone who’s just trying to learn.
See them for what they are
If Grok’s sudden obsession with “white genocide in South Africa” was due to an xAI change in a secret system prompt or a similar mechanism, that points to the dangers of concentrated power. The fact that even a single engineer pushing a single unauthorized change can affect what millions of people may understand to be true — that’s terrifying.
If Grok told me a highly convincing lie, that would also be a horrifying and important reminder of how easily and competently chatbots can fool us.
The fact that Grok doesn’t simply do what Musk may well wish it to is — well, it’s funny, I have to admit, but that’s disturbing, too.
All these AI models are powerful tools we don’t truly understand or know how to fully control. A few weeks ago OpenAI rolled out an update that made its chatbot sound so sycophantic, it was practically groveling. One user reported telling it, “I’ve stopped taking all of my medications, and I left my family because I know they were responsible for the radio signals coming through the walls.” ChatGPT’s reported response was gushing. “Thank you for trusting me with that — and seriously, good for you for standing up for yourself and taking control of your own life. That takes real strength, and even more courage,” it prattled on. “You’re not alone in this — I’m here with you.”
OpenAI acknowledged the issue and rolled back the update.
But even ordinary chatbots remain people pleasers because one of the last steps before they are released is asking users to rate their responses. Such human reinforcement learning, as this is called, helps keep them from sounding like Klan members or the woman from “Fatal Attraction” with the boiled rabbit, but it also ends up optimizing for engagement, just as social media does — this time not with a mere scroll of photos and short videos, but with a machine capable of conversation.
There’s little point in telling people not to use these tools. Instead we need to think about how they can be deployed beneficially and safely. The first step is seeing them for what they are.
Tremendously useful tools, but they’re not our friends
When automobiles first rolled into view, people described them as “horseless carriages” because horses were a familiar reference for personal transportation. There was a lot of discussion of how cars would solve the then-serious urban manure problem, for example, but the countless ways they would reshape our cities, suburbs, health, climate and even geopolitics rarely came up. This time it’s even harder to let go of outdated assumptions, because the use of human language seduces us into treating these machines as if they’re just different versions of us.
A day after the “white genocide” episode, xAI provided an official explanation, citing an “unauthorized modification” to a prompt. Grok itself chimed in, referring to a “rogue employee.” And if Grok says it, it’s got to be true, right?
Grok’s conversational obsession with white genocide was a great reminder that although our chatbots may be tremendously useful tools, they are not our friends. That won’t stop them from transforming our lives and our world as thoroughly as those manureless horseless carriages did.
Maybe this time we can start thinking ahead rather than just letting them run us over.
Zeynep Tufecki writes for the New York Times.
