From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI

Roman Beliy, Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, Michal Irani

פרסום מחקרי: פרסום בכתב עתמאמר מכנסביקורת עמיתים

73 ציטוטים ‏(Scopus)

תקציר

Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient “labeled” pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on “unlabeled” data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data. This self-supervision adapts the reconstruction network to the new input test-data, despite its deviation from the statistics of the scarce training data.

שפה מקוריתאנגלית
כתב עתAdvances in Neural Information Processing Systems
כרך32
סטטוס פרסוםפורסם - 2019
פורסם באופן חיצוניכן
אירוע33rd Annual Conference on Neural Information Processing Systems, NeurIPS 2019 - Vancouver, קנדה
משך הזמן: 8 דצמ׳ 201914 דצמ׳ 2019

טביעת אצבע

להלן מוצגים תחומי המחקר של הפרסום 'From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI'. יחד הם יוצרים טביעת אצבע ייחודית.

פורמט ציטוט ביבליוגרפי