Check your data like how social scientists check data quality before analysis. Applicable for both human panel data and simulated data.
And how it applies to both human and simulated respondents.
Pew Research Center found that 4–7% of interviews in online opt-in panels came from 'bogus' respondents — and that traditional quality checks missed most of them: 84% passed the survey's trap question and 87% passed a speeding check. In 2026, Pew's methodologists warned that AI now makes it cheap to fake a respondent at scale in panels that claim to be fully human.
Artificial Societies use large networks of AI to model important groups of people. We're social and behavioral scientists, so we hold our simulations to the same standard as how we would check a human dataset before conducting social science research. A human panel can be contaminated with low-attention respondents or bot-farm participants. A carefully built simulation can be more internally coherent. The only way to tell is to look at the raw, respondent-level data.
Metrics social scientists use to check for data quality.
High-quality data has a recognisable shape. Answers spread across the options, differ from question to question, correlate into themes, and vary between different people. Good data is messy with patterns. Low-quality data — rushed, straightlined, automated or generated — can sometimes be too inconsistent, too uniform, or conversely too templated.
Individuals barely differ from each other, and no one has a consistent belief across their answers.
Messy but with pattern: clear themes, clear outliers.
Answers collapse into a few repeated shapes, loosing the noise and nuance that make us human.
Based on the checks social scientists run before trusting a dataset.
How often do respondents contradict themselves?
Needs: Questions that are relatedDo opinions differ by gender, generation, and geography realistically?
Needs: Demographic columnsDo opinions form thematic structures organically?
Needs: Several rating-scale questionsDoes the free text have depth and nuance?
Needs: At least one free-text columnA short report you can circulate.
A score and a bandHigh quality, mixed signals, low quality / unlikely human, or not assessable.
CoverageHow much of the check could run on your file. A file with no free text can't be scored on open text richness, so we report it as skipped to help contextualise the score.
FindingsOne per measure explaining how to interpret the metric, or reason why the check was skipped.
CaveatsWhat we couldn't measure, and a note that this checks internal quality, not external quality or source of data.
Caveat: This report checks the dataset's internal quality, not its accuracy or where the data came from.
The more of your file arrives intact, the more we can measure.
One column per question. We are not able to measure these metrics based on crosstab or topline summaries.
Free text unlocks open text richness checks, and demographics unlock subgroup diversity checks.
Remember to remove names, emails, phone numbers, addresses and panellist IDs first.
Get results within two working days.
Artificial Societies is SOC 2 certified and has strict GDPR compliance. We use your file to run this check and send you the report. Your file will never be shared outside Artificial Societies, and can be deleted on request. See our privacy policy and DPA.
These are the same quality checks we hold our own simulations to. How we build and evaluate them is written up in our method and evaluation.