Convolutional neural network-based encoding and decoding of visual object recognition in space and time
Number of pages
SourceNeuroImage, 180, Part A, (2018), pp. 253-266
Article / Letter to editor
Display more detailsDisplay less details
SW OZ DCC AI
PI Group Predictive Brain
PI Group Neurobiology of Language
iss. Part A
Subject110 000 Neurocognition of Language; 180 000 Predictive Brain; Cognitive artificial intelligence; DI-BCB_DCC_Theme 4: Brain Networks and Neuronal Communication
Representations learned by deep convolutional neural networks (CNNs) for object recognition are a widely investigated model of the processing hierarchy in the human visual system. Using functional magnetic resonance imaging, CNN representations of visual stimuli have previously been shown to correspond to processing stages in the ventral and dorsal streams of the visual system. Whether this correspondence between models and brain signals also holds for activity acquired at high temporal resolution has been explored less exhaustively. Here, we addressed this question by combining CNN-based encoding models with magnetoencephalography (MEG). Human participants passively viewed 1,000 images of objects while MEG signals were acquired. We modelled their high temporal resolution source-reconstructed cortical activity with CNNs, and observed a feed-forward sweep across the visual hierarchy between 75 and 200 ms after stimulus onset. This spatiotemporal cascade was captured by the network layer representations, where the increasingly abstract stimulus representation in the hierarchical network model was reflected in different parts of the visual cortex, following the visual ventral stream. We further validated the accuracy of our encoding model by decoding stimulus identity in a left-out validation set of viewed objects, achieving state-of-the-art decoding accuracy.
Upload full text
Use your RU credentials (u/z-number and password) to log in with SURFconext to upload a file for processing by the repository team.