AI Not Ready to Replace Humans in Behavior Studies

Replacing human subjects with AI surrogates, or digital twins, is something some social scientists would like to do. Human participants can be costly, get tired easily, and may experience stress during studies. However, a recent study suggests that AI twins designed to copy individual behavior might actually twist their views, creating what researchers call a “funhouse mirror” effect.

“There’s still hope,” says Olivier Toubia, a computational social scientist at Columbia Business School in New York City. “But overall, the results were somewhat disappointing.”

In a previous project reported in Marketing Science, Toubia and his team gathered data from over 2,000 people across the United States. These participants answered more than 500 questions about their age, ethnicity, income, education, religious practices, political views, personality traits, spending habits, math skills, and vocabulary. The questions were based on commonly used scales in psychology, economics, and business research. Participants also completed online tests to assess their thinking patterns and biases.

Since research on digital twins is still limited, the goal was to create an open-source dataset for others to use. Toubia says it has been downloaded around 25,000 times so far.

For the new study, the researchers fed all the information about each individual into a large language model and asked the AI to respond as if it were that person. Across 19 experiments in social science, they tested how individuals and their AI twins reacted to various scenarios, such as donations to political candidates and opinions on algorithmic hiring. The results showed that the AI twins often did not match human behavior accurately, highlighting challenges in using them for behavioral research.