Linking Theory of Mind in human-agent interactions to validated evaluations: Can explicit questionnaires measure implicit behaviour?
New York, NY : Association for Computing Machinery (ACM)
InProceedings of the 21th ACM International Conference on Intelligent Virtual Agents, pp. 120-127
IVA '21: 21th ACM International Conference on Intelligent Virtual Agents (Japan, 14-17 September, 2021)
Article in monograph or in proceedings
Display more detailsDisplay less details
SW OZ BSI CW
Proceedings of the 21th ACM International Conference on Intelligent Virtual Agents
SubjectCommunication and Media
There is a new crisis emerging in human-agent interaction research: Instead of using validated questionnaires, individual studies are creating new questionnaires that claim to measure identical constructs. This makes replication studies and comparisons between studies near to impossible. In turn, meta-analyses to determine which characteristics are important to create agents that the user experiences as being intelligent are difficult to conduct. As part of the attempt to battle this crisis, in this current paper, we suggest the use of a Theory of Mind task to measure the implicit social behaviour users exhibit towards a virtual agent. In a two-part study, we present findings that suggest that participants conduct this Theory of Mind task as expected: participants adapt towards our virtual agent more than when they conduct the task alone. We additionally present preliminary results correlating performance in the Theory of Mind task to validated constructs. Unfortunately, our current results do not correlate significantly to the existing constructs. Data-collection is ongoing and hence no firm conclusions can be made about this second set of results. However, our data suggest that it is important to become aware that the existing validated constructs used in HCI research may not be tapping into what the researchers assume, and hence provide a basis for important discussions about these implications.
This item appears in the following Collection(s)
- Academic publications 
Upload full text
Use your RU credentials (u/z-number and password) to log in with SURFconext to upload a file for processing by the repository team.