Using-Large-Language-Models-to-Replicate-Human-Subject-Studies

KO | EN

gitlite — search

#javascript #python #hacktoberfest #golang #go #machine-learning #rust #linux #docker #nodejs #windows #kubernetes #security #open-source

Using-Large-Language-Models-to-Replicate-Human-Subject-Studies

(★ 12)

Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

Using-Large-Language-Models-to-Replicate-Human-Subject-Studies Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

*Authors:* Gati Aher, Rosa I. Arriaga, Adam Tauman Kalai

```
@inproceedings{turingExp22,
  title={Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies},
  author={Aher, Gati V and Arriaga, Rosa I and Kalai, Adam Tauman},
  booktitle={Proceedings of the 40th International Conference on Machine Learning (ICML)},
  year={2023},
  url={https://arxiv.org/abs/2208.10264},
  organization={PMLR}
}
```

Submitted to [arXiv](https://arxiv.org/abs/2208.10264) on August 18, 2022.

*Abstract:* We introduce a new type of test, called a Turing Experiment (TE), for evaluating to what extent a given language model, such as GPT models, can simulate different aspects of human behavior. A TE can also reveal consistent distortions in a language model's simulation of a specific human behavior. Unlike the Turing Test, which involves simulating a single arbitrary individual, a TE requires simulating a representative sample of participants in human subject research. We carry out TEs that attempt to replicate well-established findings from prior studies. We design a methodology for simulating TEs and illustrate its use to compare how well different language models are able to reproduce classic economic, psycholinguistic, and social psychology experiments: Ultimatum Game, Garden Path Sentences, Milgram Shock Experiment, and Wisdom of Crowds. In the first three TEs, the existing findings were replicated using recent models, while the last TE reveals a "hyper-accuracy distortion" present in some language models (including ChatGPT and GPT-4), which could affect downstream applications in education and the arts.

*Keywords:* Turing Test, Large Language Models, Evaluation Metrics

---

Code: https://github.com/microsoft/turing-experiments/tree/main

Using-Large-Language-Models-to-Replicate-Human-Subject-Studies

Using-Large-Language-Models-to-Replicate-Human-Subject-Studies Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Using-Large-Language-Models-to-Replicate-Human-Subject-Studies

Using-Large-Language-Models-to-Replicate-Human-Subject-Studies Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README