AI Survey Respondents Miss Real Opinion by 12 Points

Pew Research Center published Can AI Stand In for Human Survey-Takers? Not Really on 30 September 2026. It is a methodological experiment rather than an opinion poll. Pew built an AI "digital twin" for each of thousands of real members of its American Trends Panel, gave each twin that person's demographic profile and dozens of their previously recorded attitudes, and had it complete the same questionnaires the humans had just answered. Comparing the two sets of results shows how far these "synthetic samples", a method Pew says is becoming more widely used in the industry, drift from what real people actually think.

The answer is: a long way. Across nearly 300 questions the AI estimates differed from the human ones by an average of 12 percentage points, and by more than 15 points on around 28% of questions. No demographic group came in under 12 points, and the model performed especially poorly for Republicans, Black adults, people who had not attended college and people who use the internet rarely or not at all. The synthetic public was also tidier than the real one: 47% of questions had at least one answer option that no AI respondent chose, the AI said "not sure" about a quarter as often as people do, and it answered around 80% of factual knowledge questions correctly against about half for real panellists.

Two findings stand out for anyone who commissions research. First, the model missed badly on topics that had recently become news. On data centres, which we covered from Pew's human polling in our report on US data centre sentiment, just 3% of AI respondents said they had heard a lot about them, against 25% of real adults. Second, the choice of model changed the story. Run on GPT-5.1, the synthetic public held more extreme views than the real one; run on Claude Opus 4.6, it was more middle-of-the-road. Opus had the lowest error of the three models Pew tested, but neither it nor GPT-5.1 produced a reliable picture of the public.

Pew replicated three waves of its American Trends Panel, a nationally representative panel of randomly selected US adults: Wave 185 (20 to 25 January 2026, 8,512 respondents, of whom a weighted subsample of 6,700 was replicated with Opus because of cost), Wave 190 (23 to 30 March 2026, 3,398 of 3,507 respondents) and Wave 192 (20 to 26 April 2026, 4,981 of 5,103 respondents). Only panellists who had also completed Pew's 2025 political typology survey were included. Each AI twin received the person's demographics, 71 further variables from that typology survey and a short "expert reflection" on the profile written by GPT-5.1, then answered the whole questionnaire in order, with the same split forms, randomisation and language as its human counterpart. Replications started the day after each human wave closed, except Wave 185, which was used for configuration testing. Survey weights were applied to the synthetic sample in the same way as to the human data.

Accuracy was measured as the average absolute percentage-point gap between human and AI estimates across each question's answer options. Pew chose its configuration in rounds: reasoning level made no measurable difference, adding the extended profile cut error from 16 to 13.7 points and the expert reflection to 13.1, and on the test subsample GPT-5 nano scored 17.4, GPT-5.1 13.3 and Claude Opus 4.6 11.4. Two caveats apply. The questions were mostly about US politics and current affairs rather than brands or purchases, and Pew notes that newer models or methods could produce different patterns. It is a test of AI respondents only, not of other uses of AI in research, and it says nothing about the cost of reaching real people, which our report on how a £10 incentive lifted ONS survey response among under-30s shows can be modest.