标签归档:fMRI

Simultaneously uncovering the patterns of brain regions involved in different story reading Subprocesses

Simultaneously Uncovering the Patterns of Brain Regions Involved in Different  Story Reading Subprocesses

This Story understanding involves many perceptual and cognitive subprocesses, from perceiving individual words, to parsing sentences, to understanding the relationships among the story characters. We present an integrated computational model of reading that incorporates these and additional subprocesses, simultaneously discovering their fMRI signatures. Our model predicts the fMRI activity associated with reading arbitrary text passages, well enough to distinguish which of two story segments is being read with 74% accuracy. This approach is the first to simultaneously track diverse reading subprocesses during complex story processing and predict the detailed neural representation of diverse story features, ranging from visual word properties to the mention of different story characters and different actions they perform. We construct brain representation maps that replicate many results from a wide range of classical studies that focus each on one aspect of language processing and offer new insights on which type of information is processed by different areas involved in language processing. Additionally, this approach is promising for studying individual differences: it can be used to create single subject maps that may potentially be used to measure reading comprehension and diagnose reading disorders. This work was supported by the National Science Foundation (nsf.gov, 0835797, TM); the National Institute of Child Health and human Development (nichd.nih.gov, 5R01HD075328, TM); and the Rothberg Brain Imaging Award (http://www. cmu.edu/news/archive/2011/July/july7- rothbergawards.shtml, LW AF). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

理解故事这个任务包括了许多感知和认知上的流程,例如感知独立的单词,组合成句子以及理解故事中人物的关系。我们提出了一个计算模型,模型可以合并上述流程,同时发现他们的fMRI特征。我们的模型通过让受试者朗读随机的文段所获得的fMRI活动进行预测。并获得了文段分类的74%的准确率。这个方法是第一个可以同时追踪复杂故事阅读流程的方法,它可以预测关于多样的故事内容的细节神经表示,包括当提到不同故事角色时的视觉-语言特征和他们的不同动作。我们根据之前的研究构造脑部的表示图,它专注于语言处理,并且为其他领域能够运用提供了支持。另外,这个方法研究了个体差异,可以用作创建独立的个体样本,这些样本可能可以用作诊断阅读障碍。

Generic decoding of seen and imagined objects using hierarchical visual features

Generic decoding of seen and imagined objects using hierarchical visual  features | Nature Communications

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.

https://www.nature.com/articles/ncomms15037

目标识别是人类和机器视觉的主要功能。虽然对于看见或者想象目标时的大脑解码任务已经可以做到,但是预测性能还受制于训练样本。我们提出了一个可以作用于任意目标的解码方法,它利用机器视觉的机制:一个目标在不同层级的模型中会以无关的特征形式呈现。我们发现从深度卷积神经网络中获得的视觉特征也可以由fMRI图像中预测获得,对应的低-低,高-高层级特征可以获得更高的预测精度。预测出来的特征还能够用于识别看到或者想到的目标(甚至是训练时没有用到的类别)。试验结果更加证明了视觉特征是从高到低等级渐进表示的。我们的实验还证明了人类和机器视觉拥有相同的作用机制。

BOLD5000, a public fMRI dataset while viewing 5000 visual images

figure2

Vision science, particularly machine vision, has been revolutionized by introducing large-scale image datasets and statistical learning approaches. Yet, human neuroimaging studies of visual perception still rely on small numbers of images (around 100) due to time-constrained experimental procedures. To apply statistical learning approaches that include neuroscience, the number of images used in neuroimaging must be significantly increased. We present BOLD5000, a human functional MRI (fMRI) study that includes almost 5,000 distinct images depicting real-world scenes. Beyond dramatically increasing image dataset size relative to prior fMRI studies, BOLD5000 also accounts for image diversity, overlapping with standard computer vision datasets by incorporating images from the Scene UNderstanding (SUN), Common Objects in Context (COCO), and ImageNet datasets. The scale and diversity of these image datasets, combined with a slow event-related fMRI design, enables fine-grained exploration into the neural representation of a wide range of visual features, categories, and semantics. Concurrently, BOLD5000 brings us closer to realizing Marr’s dream of a singular vision science–the intertwined study of biological and computer vision.

https://bold5000.github.io

视觉科学,尤其是机器视觉,已经因为大规模图像数据集和统计学习方法地提出而带来革命性的变化。但是目前为止,对于人类视觉感知的神经图像研究依然依赖着少量的数据(常常是100张图片左右),这是因为实验过程受到时间的限制。为了将统计学习的方法介绍到神经科学领域,数据量必须有一个大的提升。我们提出了BOLD5000,一个人类fMRI的研究,它包括接近5000张不同的描述真实场景的图像。相对于之前的fMRI数据集,BOLD5000还更加注重图像的多样性,很多的图像与标准图像数据集有交集,例如SUN,COCO等。除了注重规模和多样性,BOLD5000还结合了慢速事件相关的fMRI设计,这使得在丰富视觉特征和种类以及语义上研究神经表达成为可能。目前,BOLD5000使得我们与Marr的梦想变得更近了:将生物和计算机世界放在一起研究。

Vox2Vox: 3D-GAN for Brain Tumour Segmentation

Vox2Vox] Vox2Vox: 3D-GAN for Brain Tumour Segmentation · Issue #5 ·  e4exp/paper_manager · GitHub

Gliomas are the most common primary brain malignancies, with different degrees of aggressiveness, variable prognosis and various heterogeneous histological sub-regions, i.e., peritumoral edema, necrotic core, enhancing and non-enhancing tumor core. Although both these brain tumour types can easily be detected using multi-modal MRI, and exact them doing image segmentation is a challenging task. Hence, using the data provided by the BraTS Challenge 2020, we propose a 3D volume-to-volume Generative Adversarial Network for segmentation of brain tumours. The model, called Vox2Vox, generates realistic segmentation outputs from multi-channel 3D MR images, detecting the whole, core and enhancing tumor with median values of 93.39%, 92.50%, and 87.16% as dice scores and 2.44mm, 2.23mm, and 1.73mm for Hausdorff distance 95 percentile for the training dataset, and 91.75%, 88.13%, and 85.87% and 3.0mm, 3.74mm, and 2.23mm for the validation dataset, after ensembling 10 Vox2Vox models obtained with a 10-fold cross-validation.

https://arxiv.org/pdf/2003.13653.pdf

神经胶质瘤是一种最常见的脑部恶性肿瘤,它的特点是多变的恶性程度,难以预测的预后,以及异构的子结构,例如,瘤边水肿,坏死核心,以及在发展的和非发展的肿瘤核心。虽然这些脑部肿瘤可以轻易的被多模态MRI发现,但是对它们进行精确分割却是一个挑战。因此,我们提出了一种3D体到体的生成对抗网络并且在BraTS Challenge 2020数据上进行了测试。模型的名称为Vox2Vox,它可以在不同条件下为3D MR图像生成逼真的分割结果,检测全体、核心或者正在发展的肿瘤核心。

Inducing brain-relevant bias in natural language processing models

Progress in natural language processing (NLP) models that estimate representations of word sequences has recently been leveraged to improve the understanding of language processing in the brain. However, these models have not been specifically designed to capture the way the brain represents language meaning. We hypothesize that fine-tuning these models to predict recordings of brain activity of people reading text will lead to representations that encode more brain-activity-relevant language information. We demonstrate that a version of BERT, a recently introduced and powerful language model, can improve the prediction of brain activity after fine-tuning. We show that the relationship between language and brain activity learned by BERT during this fine-tuning transfers across multiple participants. We also show that, for some participants, the fine-tuned representations learned from both magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) are better for predicting fMRI than the representations learned from fMRI alone, indicating that the learned representations capture brain-activity-relevant information that is not simply an artifact of the modality. While changes to language representations help the model predict brain activity, they also do not harm the model’s ability to perform downstream NLP tasks. Our findings are notable for research on language understanding in the brain.

http://papers.nips.cc/paper/9559-inducing-brain-relevant-bias-in-natural-language-processing-models

自然语言处理模型对于语言序列预测的处理流程已经被认为对于理解大脑语言处理的流程有重要帮助。但是,这些模型并没有被特别为了捕捉脑部表示语言含义流程而设计。我们提供一个假设:利用测试者在朗读时的脑部活动记录fine-tuning自然语言处理模型可以使得这些模型学习到更多脑部活动相关的语言信息。我们提出了一个在fine-tuning后可以提高预测脑部活动BERT的变种,BERT是一种最近提出的性能强大的语言模型。我们通过MEG和fMRI多域数据的实验揭示了语言和大脑活动的相关性,并且说明了语言模型学习到的特征是捕捉到脑部活动相关信息的,而不是无意义的。当利用语言表示去预测脑部活动的时候,模型依然能够很好地执行downstream的自然语言处理任务。