This study asks whether person-fit statistics, a set of Item Response Theory tools that check how well an examinee's answer pattern matches what the model expects, can tell generative AI answers apart from human answers on multiple-choice tests. The authors compared chatbot responses with those of about 1000 human test-takers in each of 2 real assessments, a high-school chemistry test and a national university entrance exam.
- Person-fit statistics found significant differences between human answer patterns and those produced by advanced chatbots, with ChatGPT, Claude and Gemini among those tested.
- The chatbots showed up as aberrant respondents, meaning their answer patterns departed from what the model predicts for a typical examinee.
- Different chatbots produced significantly different response patterns from one another.
- Newer versions of generative AI performed better on the tests, and their answer patterns also moved closer to those of human examinees.
The authors argue that chatbots should be treated as a mixed group of separate intelligences, since their patterns differ, and that multiple-choice tests, which detection work has largely passed over as too hard to check, can be examined with existing psychometric methods. They conclude that Item Response Theory gives a sound framework for describing and separating human and AI response patterns on these tests, and they present the work as a theoretical and empirical base for that use. Their finding that newer models look more human-like in their answer patterns suggests the gap these statistics pick up may narrow as the tools improve.
