Abstract
We use visual information to build up complex and meaningful mental models of ourselves and our environments. However, it is largely unknown how expectations derived from these complex mental models influence eye movements. To address this question we here recorded gaze behaviour of adult human volunteers as they were watching picture stories. These were presented either in a meaningful order, or a scrambled order. Each story was presented repeatedly, allowing to assess how narrative knowledge influences subsequent visual sampling. We tested how a state-of-the-art model of gaze behaviour based on visual saliency, DeepGaze II, accounts for the observed gaze behaviour. Our preliminary results indicate that the model captures gaze behaviour better when images are presented in a scrambled than intact order, and during first rather than subsequent viewing. This suggests that as one acquires expectations about the narrative, either from world knowledge or from previous experience, sampling behaviour starts to deviate from what can be predicted based on a visual saliency model. We speculate that this is caused by a shift towards information sampling based on meaning-based expectations. We are currently testing this hypothesis by quantifying meaning-based expectations in stories using language modelling (GPT-2).
