TractoFormer: A Novel Fiber-level Whole Brain Tractography Analysis Framework Using Spectral Embedding and Vision Transformers

[Audio] Hello everyone, in this presentation, we will introduce our work on, TractoFormer, A Novel Fiber-level Whole Brain Tractography Analysis Framework Using Spectral Embedding and Vision Transformers..

[Audio] Diffusion MRI tractography is an advanced imaging technique that enables in vivo reconstruction of the brain's white matter ( WM) connections, and it provides an important tool for quantitative mapping of the brain's connectivity to study the brain in health and disease. Currently, there are several challenges when performing disease classification using whole brain tractography data and machine learning. First, performing whole brain tractography on one individual subject can generate hundreds of thousands, or even millions, of fiber streamlines. Defining a good data representation of tractography for machine learning is still an open challenge, especially at the fiber level. The second challenge is the limited sample size, i.e., number of subjects, of many dMRI datasets. Small sample sizes limit the use of recently proposed advanced learning techniques such as Vision Transformers, which are highly accurate but usually require a large number of samples to avoid overfitting. The third challenge is about result interpretability, where deep learning methods for neuroimaging should be able to pinpoint location(s) in the brain that are predictive of disease or affected by disease.

[Audio] In this paper, we propose a novel parcellation-free WBT analysis framework, TractoFormer, that leverages tractography information at the level of individual fiber streamlines and provides a natural mechanism for interpretation of results using the self-attention scheme of vision transformers. TractoFormer includes two main contributions. First, we propose a novel 2D image representation of WBT, referred to as TractoEmbedding, based on a spectral embedding of fibers from tractography. Second, we propose a ViT- based network that performs effective and interpretable group classification..

[Audio] The TractoEmbedding process includes three major steps, as illustrated in this figure. First, we perform spectral embedding to represent each fiber in the whole brain tractography data as a point in a latent space, where nearby points correspond to spatially proximate fibers. Second, the coordinates of each fiber of the tractography data are discretized onto a 2D grid for creation of an image. Each dimension of the embedding coordinate vector corresponds to the eigenvectors of the affinity matrix sorted in descending order. In our study, we choose the first two dimensions for each point and discretize them onto a 2D embedding gird. Third, we map the measure of interest associated with each fiber to the corresponding pixel on the embedding grid as its intensity value. This generates a 2D image, i.e., the TractoEmbedding image. When multiple fibers that are spatially proximate are mapped to the same voxel, we can compute summary statistics from these fibers, such as max, min, and mean (mean is used in our experiments)..

[Audio] Then, based on the TractoEmbedding images, we propose ViT-based Framework for group classification, as shown in the figure. Our design aims to address the aforementioned challenges of sample size and interpretability. First, we leverage the multi-sample data augmentation to reduce the known overfitting issue of ViTs on small sample size datasets. One important feature of this framework is that multiple TractoEmbedding images can be generated from one subject by performing random downsampling of the whole brain tractography data. In this way, this provides a natural data augmentation strategy to increase sample size. Second, we leverage the self-attention scheme in ViT to identify discriminative fibers that are most useful to differentiate between groups. The interpretation of the ViT attention maps is aided by our proposed multi-channel architecture, which can enable inspection of the independent contributions of different brain regions. As a result, we are able to identify the fibers that are discriminative for result interpretation..

[Audio] We first perform an experiment using synthetic dataset. The goal is to provide a proof-of-concept evaluation to assess if the proposed method can successfully classify groups with true white matter differences and identify the fibers with group differences in the tractography data for interpretation. To do so, we create a realistic synthetic dataset with true group differences in the corticospinal tract. As for results, TractoFormer achieved, as expected, 100% group classification accuracy because of the added synthetic feature changes to G2. The ientified discriminative fibers are generally similar to the CST fibers with synthetic changes..

[Audio] In this experiment, we performed disease classification between heathy controls and schizophrenia. The goal is to evaluate the proposed TractoFormer in a real neuroscientific application for brain disease classification. The evaluation dataset includes data from 103 healthy controls and 47 schizophrenia patients from the Consortium for Neuropsychiatric Phenomics database. We compare our method with three baseline methods. As shown in Table 1, our method gave the best overall result, with a mean accuracy of 0.849 and a mean F1 of 0.770. The figure gives a visualization of the discriminative fibers from group-wise and subject-specific attention maps, showing that the superficial fibers in the frontal and parietal lobes have high importance when classifying heathy controls and schizophrenia patients under study..

[Audio] In this paper, we present a novel parcellation-free whole brain tractography analysis framework, TractoFormer, which leverages tractography information at the level of individual fiber streamlines and provides a natural mechanism for interpretation of results using attention. We propose random sampling of tractography as an effective data augmentation strategy for small sample size whole brain tractography datasets. Overall, TractoFormer suggests the potential for deep learning analysis of whole brain tractography represented as images..

[Audio] Thank you for your attention.. Thank you!.