Cortical network responses map onto data-driven features that capture visual semantics of movie fragments
Publication year
2020Author(s)
Number of pages
21 p.
Source
Scientific Reports, 10, (2020), article 12077ISSN
Publication type
Article / Letter to editor

Display more detailsDisplay less details
Organization
SW OZ DCC AI
PI Group Neurobiology of Language
Journal title
Scientific Reports
Volume
vol. 10
Languages used
English (eng)
Subject
110 000 Neurocognition of Language; Cognitive artificial intelligence; Language in InteractionAbstract
Research on how the human brain extracts meaning from sensory input relies in principle on methodological reductionism. In the present study, we adopt a more holistic approach by modeling the cortical responses to semantic information that was extracted from the visual stream of a feature film, employing artificial neural network models. Advances in both computer vision and natural language processing were utilized to extract the semantic representations from the film by combining perceptual and linguistic information. We tested whether these representations were useful in studying the human brain data. To this end, we collected electrocorticography responses to a short movie from 37 subjects and fitted their cortical patterns across multiple regions using the semantic components extracted from film frames. We found that individual semantic components reflected fundamental semantic distinctions in the visual input, such as presence or absence of people, human movement, landscape scenes, human faces, etc. Moreover, each semantic component mapped onto a distinct functional cortical network involving high-level cognitive regions in occipitotemporal, frontal and parietal cortices. The present work demonstrates the potential of the data-driven methods from information processing fields to explain patterns of cortical responses, and contributes to the overall discussion about the encoding of high-level perceptual information in the human brain.
Subsidient
NWO (Grant code:info:eu-repo/grantAgreement/NWO/Gravitation/024.001.006)
This item appears in the following Collection(s)
- Academic publications [203793]
- Donders Centre for Cognitive Neuroimaging [3390]
- Electronic publications [102109]
- Faculty of Social Sciences [27292]
- Open Access publications [70806]
Upload full text
Use your RU credentials (u/z-number and password) to log in with SURFconext to upload a file for processing by the repository team.