Abstract:Reconstructing 3D visuals from functional Magnetic Resonance Imaging (fMRI) data, introduced as Recon3DMind in our conference work, is of significant interest to both cognitive neuroscience and computer vision. To advance this task, we present the fMRI-3D dataset, which includes data from 15 participants and showcases a total of 4768 3D objects. The dataset comprises two components: fMRI-Shape, previously introduced and accessible at <a class="link-external link-https" href="https://huggingface.co/datasets/Fudan-fMRI/fMRI-Shape" rel="external noopener nofollow">this https URL</a>, and fMRI-Objaverse, proposed in this paper and available at <a class="link-external link-https" href="https://huggingface.co/datasets/Fudan-fMRI/fMRI-Objaverse" rel="external noopener nofollow">this https URL</a>. fMRI-Objaverse includes data from 5 subjects, 4 of whom are also part of the Core set in fMRI-Shape, with each subject viewing 3142 3D objects across 117 categories, all accompanied by text captions. This significantly enhances the diversity and potential applications of the dataset. Additionally, we propose MinD-3D, a novel framework designed to decode 3D visual information from fMRI signals. The framework first extracts and aggregates features from fMRI data using a neuro-fusion encoder, then employs a feature-bridge diffusion model to generate visual features, and finally reconstructs the 3D object using a generative transformer decoder. We establish new benchmarks by designing metrics at both semantic and structural levels to evaluate model performance. Furthermore, we assess our model's effectiveness in an Out-of-Distribution setting and analyze the attribution of the extracted features and the visual ROIs in fMRI signals. Our experiments demonstrate that MinD-3D not only reconstructs 3D objects with high semantic and spatial accuracy but also deepens our understanding of how human brain processes 3D visual information. Project page at: <a class="link-external link-https" href="https://jianxgao.github.io/MinD-3D" rel="external noopener nofollow">this https URL</a>.

3D-COCO: extension of MS-COCO dataset for image detection and 3D reconstruction modules

COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images

CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection

COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval

Benchmarking Object Detectors with COCO: A New Path Forward

COCO-Occ: A Benchmark for Occluded Panoptic Segmentation and Image Understanding

Benchmarking a Benchmark: How Reliable is MS-COCO?

OmniNOCS: A unified NOCS dataset and model for 3D lifting of 2D objects

Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection

CD-COCO: A Versatile Complex Distorted COCO Database for Scene-Context-Aware Computer Vision

Towards Robust Robot 3D Perception in Urban Environments: The UT Campus Object Dataset

ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Dataset

Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image

fMRI-3D: A Comprehensive Dataset for Enhancing fMRI-based 3D Reconstruction

OCM3D: Object-Centric Monocular 3D Object Detection

SR-CACO-2: A Dataset for Confocal Fluorescence Microscopy Image Super-Resolution

Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework

ObjectNet3D: A Large Scale Database for 3D Object Recognition

COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts

Category-level Object Detection, Pose Estimation and Reconstruction from Stereo Images