# Theory of Mind Performance of Large Language Models

How GPT-3.5 and GPT-4 performed on theory-of-mind tasks in Turkish and English.

Burcu Ünlütabak, Onur Bal

2025 · Journal article · Computer Speech & Language

DOI: https://doi.org/10.1016/j.csl.2024.101698

## Overview

Can GPT-3.5 and GPT-4 track what a character believes when that belief differs from reality? Burcu Ünlütabak and Onur Bal tested both models in Turkish and English. First-order tasks asked what a character believed. Second-order tasks asked what one character thought another character believed.

## How we tested the models

Three experiments replicated and extended standard false-belief scenarios. The researchers paired sentence completion with open-ended questions in the same conversation so they could compare the models’ answers with their explanations. The Turkish tasks also let the team study wording that uses verbs such as _sanmak_ and _zannetmek_, which can signal a mistaken belief.

## What we found

GPT-4 performed better than GPT-3.5 overall, and both models generally performed better in English than in Turkish. The models answered some standard tasks correctly but lost accuracy after small changes to the stories. The results show how sensitive these model versions were to the design and wording of the tasks.

## Research contributions

Ünlütabak developed the research question and supervised the study. Bal built the Python and Jupyter experiment pipeline. Both authors contributed to methodology, data curation, analysis, and visualization. The article includes the full contribution statement.

## Read the paper and materials

Published in _Computer Speech & Language_, volume 89, article 101698.

[Read the full paper (PDF)](https://onurbal.com/files/academics/theory-of-mind-performance.pdf) · [Publisher record](https://doi.org/10.1016/j.csl.2024.101698) · [Prompts, code, and model outputs](https://github.com/onurbal101/gpt_tom)

## Related work

[Explore our research themes](/research.md) · [Children’s questions to voice assistants](/research/children-voice-assistants.md) · [AI trust and information seeking](/research/ai-trust.md)

## Citation

Ünlütabak, B., & Bal, O. (2025). Theory of mind performance of large language models: A comparative analysis of Turkish and English. Computer Speech & Language, 89, Article 101698. https://doi.org/10.1016/j.csl.2024.101698

### BibTeX

```bibtex
@article{unlutabak2025theory,
  title = {Theory of Mind Performance of Large Language Models: A Comparative Analysis of Turkish and English},
  author = {Ünlütabak, Burcu and Bal, Onur},
  journal = {Computer Speech & Language},
  volume = {89},
  pages = {101698},
  year = {2025},
  doi = {10.1016/j.csl.2024.101698}
}
```

---

[Curtainplate Laboratories](/index.md) · [Research](/research.md) · [People](/people.md) · [Publications](/publications.md) · [About](/about.md) · [Contact](/contact.md) · [Join us](/join.md) · [Take part](/participate.md) · [Privacy](/privacy.md)