Multiple peritoneal calcifying fibrous tumors revealed by ischemic colitis.

R. Farah,S. Dimet,A. Bidault,Cedric Benbraheim,N. Ndiaye,T. Lazure,F. Gayral,C. Buffet,G. Pelletier

DOI: https://doi.org/10.1016/J.ANNDIAGPATH.2006.06.004

IF: 2.134

2007-12-01

Annals of Diagnostic Pathology

Abstract:

What problem does this paper attempt to address?

Implementing vision transformer for classifying 2D biomedical images

Arindam Halder,Sanghita Gharami,Priyangshu Sadhu,Pawan Kumar Singh,Marcin Woźniak,Muhammad Fazal Ijaz

DOI: https://doi.org/10.1038/s41598-024-63094-9

IF: 4.6

2024-06-01

Scientific Reports

Abstract:In recent years, the growth spurt of medical imaging data has led to the development of various machine learning algorithms for various healthcare applications. The MedMNISTv2 dataset, a comprehensive benchmark for 2D biomedical image classification, encompasses diverse medical imaging modalities such as Fundus Camera, Breast Ultrasound, Colon Pathology, Blood Cell Microscope etc. Highly accurate classifications performed on these datasets is crucial for identification of various diseases and determining the course of treatment. This research paper presents a comprehensive analysis of four subsets within the MedMNISTv2 dataset: BloodMNIST, BreastMNIST, PathMNIST and RetinaMNIST. Each of these selected datasets is of diverse data modalities and comes with various sample sizes, and have been selected to analyze the efficiency of the model against diverse data modalities. The study explores the idea of assessing the Vision Transformer Model's ability to capture intricate patterns and features crucial for these medical image classification and thereby transcend the benchmark metrics substantially. The methodology includes pre-processing the input images which is followed by training the ViT-base-patch16-224 model on the mentioned datasets. The performance of the model is assessed using key metrices and by comparing the classification accuracies achieved with the benchmark accuracies. With the assistance of ViT, the new benchmarks achieved for BloodMNIST, BreastMNIST, PathMNIST and RetinaMNIST are 97.90%, 90.38%, 94.62% and 57%, respectively. The study highlights the promise of Vision transformer models in medical image analysis, preparing the way for their adoption and further exploration in healthcare applications, aiming to enhance diagnostic accuracy and assist medical professionals in clinical decision-making.

multidisciplinary sciences
MedViT: A robust vision transformer for generalized medical image classification

Omid Nejati Manzari,Hamid Ahmadabadi,Hossein Kashiani,Shahriar B. Shokouhi,Ahmad Ayatollahi

DOI: https://doi.org/10.1016/j.compbiomed.2023.106791

IF: 7.7

2023-05-01

Computers in Biology and Medicine

Abstract:Convolutional Neural Networks (CNNs) have advanced existing medical systems for automatic disease diagnosis. However, there are still concerns about the reliability of deep medical diagnosis systems against the potential threats of adversarial attacks since inaccurate diagnosis could lead to disastrous consequences in the safety realm. In this study, we propose a highly robust yet efficient CNN-Transformer hybrid model which is equipped with the locality of CNNs as well as the global connectivity of vision Transformers. To mitigate the high quadratic complexity of the self-attention mechanism while jointly attending to information in various representation subspaces, we construct our attention mechanism by means of an efficient convolution operation. Moreover, to alleviate the fragility of our Transformer model against adversarial attacks, we attempt to learn smoother decision boundaries. To this end, we augment the shape information of an image in the high-level feature space by permuting the feature mean and variance within mini-batches. With less computational complexity, our proposed hybrid model demonstrates its high robustness and generalization ability compared to the state-of-the-art studies on a large-scale collection of standardized MedMNIST-2D datasets.

engineering, biomedical,computer science, interdisciplinary applications,mathematical & computational biology,biology
Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review

Reza Azad,Amirhossein Kazerouni,Moein Heidari,Ehsan Khodapanah Aghdam,Amirali Molaei,Yiwei Jia,Abin Jose,Rijo Roy,Dorit Merhof

2023-11-05

Abstract:The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in <a class="link-external link-https" href="https://github.com/mindflow-institue/Awesome-Transformer" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
Vision Transformer for Efficient Chest X-ray and Gastrointestinal Image Classification

Smriti Regmi,Aliza Subedi,Ulas Bagci,Debesh Jha

2023-04-23

Abstract:Medical image analysis is a hot research topic because of its usefulness in different clinical applications, such as early disease diagnosis and treatment. Convolutional neural networks (CNNs) have become the de-facto standard in medical image analysis tasks because of their ability to learn complex features from the available datasets, which makes them surpass humans in many image-understanding tasks. In addition to CNNs, transformer architectures also have gained popularity for medical image analysis tasks. However, despite progress in the field, there are still potential areas for improvement. This study uses different CNNs and transformer-based methods with a wide range of data augmentation techniques. We evaluated their performance on three medical image datasets from different modalities. We evaluated and compared the performance of the vision transformer model with other state-of-the-art (SOTA) pre-trained CNN networks. For Chest X-ray, our vision transformer model achieved the highest F1 score of 0.9532, recall of 0.9533, Matthews correlation coefficient (MCC) of 0.9259, and ROC-AUC score of 0.97. Similarly, for the Kvasir dataset, we achieved an F1 score of 0.9436, recall of 0.9437, MCC of 0.9360, and ROC-AUC score of 0.97. For the Kvasir-Capsule (a large-scale VCE dataset), our ViT model achieved a weighted F1-score of 0.7156, recall of 0.7182, MCC of 0.3705, and ROC-AUC score of 0.57. We found that our transformer-based models were better or more effective than various CNN models for classifying different anatomical structures, findings, and abnormalities. Our model showed improvement over the CNN-based approaches and suggests that it could be used as a new benchmarking algorithm for algorithm development.

Image and Video Processing,Computer Vision and Pattern Recognition
Vision Transformers in Medical Computer Vision -- A Contemplative Retrospection

Arshi Parvaiz,Muhammad Anwaar Khalid,Rukhsana Zafar,Huma Ameer,Muhammad Ali,Muhammad Moazam Fraz

DOI: https://doi.org/10.48550/arXiv.2203.15269

2022-03-29

Abstract:Recent escalation in the field of computer vision underpins a huddle of algorithms with the magnificent potential to unravel the information contained within images. These computer vision algorithms are being practised in medical image analysis and are transfiguring the perception and interpretation of Imaging data. Among these algorithms, Vision Transformers are evolved as one of the most contemporary and dominant architectures that are being used in the field of computer vision. These are immensely utilized by a plenty of researchers to perform new as well as former experiments. Here, in this article we investigate the intersection of Vision Transformers and Medical images and proffered an overview of various ViTs based frameworks that are being used by different researchers in order to decipher the obstacles in Medical Computer Vision. We surveyed the application of Vision transformers in different areas of medical computer vision such as image-based disease classification, anatomical structure segmentation, registration, region-based lesion Detection, captioning, report generation, reconstruction using multiple medical imaging modalities that greatly assist in medical diagnosis and hence treatment process. Along with this, we also demystify several imaging modalities used in Medical Computer Vision. Moreover, to get more insight and deeper understanding, self-attention mechanism of transformers is also explained briefly. Conclusively, we also put some light on available data sets, adopted methodology, their performance measures, challenges and their solutions in form of discussion. We hope that this review article will open future directions for researchers in medical computer vision.

Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning
A Novel Vision Transformer with Residual in Self-attention for Biomedical Image Classification

Arun K. Sharma,Nishchal K. Verma

2023-06-05

Abstract:Biomedical image classification requires capturing of bio-informatics based on specific feature distribution. In most of such applications, there are mainly challenges due to limited availability of samples for diseased cases and imbalanced nature of dataset. This article presents the novel framework of multi-head self-attention for vision transformer (ViT) which makes capable of capturing the specific image features for classification and analysis. The proposed method uses the concept of residual connection for accumulating the best attention output in each block of multi-head attention. The proposed framework has been evaluated on two small datasets: (i) blood cell classification dataset and (ii) brain tumor detection using brain MRI images. The results show the significant improvement over traditional ViT and other convolution based state-of-the-art classification models.

Computer Vision and Pattern Recognition
MetaMed: Few-shot medical image classification using gradient-based meta-learning

Rishav Singh,Vandana Bharti,Vishal Purohit,Abhinav Kumar,Amit Kumar Singh,Sanjay Kumar Singh

DOI: https://doi.org/10.1016/j.patcog.2021.108111

IF: 8

2021-12-01

Pattern Recognition

Abstract:The occurrence of long-tailed distributions and unavailability of high-quality annotated images is a common phenomenon in medical datasets. The use of conventional Deep Learning techniques to obtain an unbiased model with high generalization accuracy for such datasets is a challenging task. Thus, we formulated a few-shot learning problem and presented a meta-learning-based "MetaMed" approach. The model presented here can adapt to rare disease classes with the availability of few images, and less compute. MetaMed is validated on three publicly accessible medical datasets – Pap smear, BreakHis, and ISIC 2018. We used advanced image augmentation techniques like CutOut, MixUp, and CutMix to overcome the problem of over-fitting. Our approach has shown promising results on all the three datasets with an accuracy of more than 70%. Inclusion of advanced augmentation techniques regularizes the model and increases the generalization capability by 2-5%. Comparative analysis of MetaMed against transfer learning demonstrated that MetaMed classifies images with a higher confidence score and on average outperforms transfer learning for 3, 5, and 10-shot tasks for both 2-way and 3-way classification.

computer science, artificial intelligence,engineering, electrical & electronic
Using Meta-Transformers for Multimodal Clinical Decision Support and Evidence-Based Medicine

Sabah Mohammed,Jinan Fiaidhi,Abel Serracin Martinez

DOI: https://doi.org/10.1101/2024.08.14.24312001

2024-08-20

Abstract:The advancements in computer vision and natural language processing are keys to thriving modern healthcare systems and its applications. Nonetheless, they have been researched and used as separate technical entities without integrating their predictive knowledge discovery when they are combined. Such integration will benefit every clinical/medical problem as they are inherently multimodal - they involve several distinct forms of data, such as images and text. However, the recent advancements in machine learning have brought these fields closer using the notion of meta-transformers. At the core of this synergy is building models that can process and relate information from multiple modalities where the raw input data from various modalities are mapped into a shared token space, allowing an encoder to extract high-level semantic features of the input data. Nerveless, the task of automatically identifying arguments in a clinical/medical text and finding their multimodal relationships remains challenging as it does not rely only on relevancy measures (e.g. how close that text to other modalities like an image) but also on the evidence supporting that relevancy. Relevancy based on evidence is a normal practice in medicine as every practice is an evidence-based. In this article we are experimenting with meta-transformers that can benefit evidence based predictions. In this article, we are experimenting with variety of fine tuned medical meta-transformers like PubmedCLIP, CLIPMD, BiomedCLIP-PubMedBERT and BioCLIP to see which one provide evidence-based relevant multimodal information. Our experimentation uses the TTi-Eval open-source platform to accommodate multimodal data embeddings. This platform simplifies the integration and evaluation of different meta-transformers models but also to variety of datasets for testing and fine tuning. Additionally, we are conducting experiments to test how relevant any multimodal prediction to the published medical literature especially those that are published by PubMed. Our experimentations revealed that the BiomedCLIP-PubMedBERT model provide more reliable evidence-based relevance compared to other models based on randomized samples from the ROCO V2 dataset or other multimodal datasets like MedCat. In this next stage of this research we are extending the use of the winning evidence-based multimodal learning model by adding components that enable medical practitioner to use this model to predict answers to clinical questions based on sound medical questioning protocol like PICO and based on standardized medical terminologies like UMLS.
Improved EATFormer: A Vision Transformer for Medical Image Classification

Yulong Shisu,Susano Mingwin,Yongshuai Wanwag,Zengqiang Chenso,Sunshin Huing

2024-03-20

Abstract:The accurate analysis of medical images is vital for diagnosing and predicting medical conditions. Traditional approaches relying on radiologists and clinicians suffer from inconsistencies and missed diagnoses. Computer-aided diagnosis systems can assist in achieving early, accurate, and efficient diagnoses. This paper presents an improved Evolutionary Algorithm-based Transformer architecture for medical image classification using Vision Transformers. The proposed EATFormer architecture combines the strengths of Convolutional Neural Networks and Vision Transformers, leveraging their ability to identify patterns in data and adapt to specific characteristics. The architecture incorporates novel components, including the Enhanced EA-based Transformer block with Feed-Forward Network, Global and Local Interaction , and Multi-Scale Region Aggregation modules. It also introduces the Modulated Deformable MSA module for dynamic modeling of irregular locations. The paper discusses the Vision Transformer (ViT) model's key features, such as patch-based processing, positional context incorporation, and Multi-Head Attention mechanism. It introduces the Multi-Scale Region Aggregation module, which aggregates information from different receptive fields to provide an inductive bias. The Global and Local Interaction module enhances the MSA-based global module by introducing a local path for extracting discriminative local information. Experimental results on the Chest X-ray and Kvasir datasets demonstrate that the proposed EATFormer significantly improves prediction speed and accuracy compared to baseline models.

Computer Vision and Pattern Recognition
Stitched vision transformer for age-related macular degeneration detection using retinal optical coherence tomography images

Mohammad Mahdi Azizi,Setareh Abhari,Hedieh Sajedi

DOI: https://doi.org/10.1371/journal.pone.0304943

IF: 3.7

2024-06-06

PLoS ONE

Abstract:Age-related macular degeneration (AMD) is an eye disease that leads to the deterioration of the central vision area of the eye and can gradually result in vision loss in elderly individuals. Early identification of this disease can significantly impact patient treatment outcomes. Furthermore, given the increasing elderly population globally, the importance of automated methods for rapidly monitoring at-risk individuals and accurately diagnosing AMD is growing daily. One standard method for diagnosing AMD is using optical coherence tomography (OCT) images as a non-invasive imaging technology. In recent years, numerous deep neural networks have been proposed for the classification of OCT images. Utilizing pre-trained neural networks can speed up model deployment in related tasks without compromising accuracy. However, most previous methods overlook the feasibility of leveraging pre-existing trained networks to search for an optimal architecture for AMD staging on a new target dataset. In this study, our objective was to achieve an optimal architecture in the efficiency-accuracy trade-off for classifying retinal OCT images. To this end, we employed pre-trained medical vision transformer (MedViT) models. MedViT combines convolutional and transformer neural networks, explicitly designed for medical image classification. Our approach involved pre-training two distinct MedViT models on a source dataset with labels identical to those in the target dataset. This pre-training was conducted in a supervised manner. Subsequently, we evaluated the performance of the pre-trained MedViT models for classifying retinal OCT images from the target Noor Eye Hospital (NEH) dataset into the normal, drusen, and choroidal neovascularization (CNV) classes in zero-shot settings and through five-fold cross-validation. Then, we proposed a stitching approach to search for an optimal model from two MedViT family models. The proposed stitching method is an efficient architecture search algorithm known as stitchable neural networks. Stitchable neural networks create a candidate model in search space for each pair of stitchable layers by inserting a linear layer between them. A pair of stitchable layers consists of layers, each selected from one input model. While stitchable neural networks had previously been tested on more extensive and general datasets, this study demonstrated that stitching networks could also be helpful in smaller medical datasets. The results of this approach indicate that when pre-trained models were available for OCT images from another dataset, it was possible to achieve a model in 100 epochs with an accuracy of over 94.9% in classifying images from the NEH dataset. The results of this study demonstrate the efficacy of stitchable neural networks as a fine-tuning method for OCT image classification. This approach not only leads to higher accuracy but also considers architecture optimization at a reasonable computational cost.

multidisciplinary sciences
Echoes of images: multi-loss network for image retrieval in vision transformers

Anshul Pundhir,Shivam Sagar,Pradeep Singh,Balasubramanian Raman

DOI: https://doi.org/10.1007/s11517-024-03055-6

2024-03-05

Medical & Biological Engineering & Computing

Abstract:This paper introduces a novel approach to enhance content-based image retrieval, validated on two benchmark datasets: ISIC-2017 and ISIC-2018. These datasets comprise skin lesion images that are crucial for innovations in skin cancer diagnosis and treatment. We advocate the use of pre-trained Vision Transformer (ViT), a relatively uncharted concept in the realm of image retrieval, particularly in medical scenarios. In contrast to the traditionally employed Convolutional Neural Networks (CNNs), our findings suggest that ViT offers a more comprehensive understanding of the image context, essential in medical imaging. We further incorporate a weighted multi-loss function, delving into various losses such as triplet loss, distillation loss, contrastive loss, and cross-entropy loss. Our exploration investigates the most resilient combination of these losses to create a robust multi-loss function, thus enhancing the robustness of the learned feature space and ameliorating the precision and recall in the retrieval process. Instead of using all the loss functions, the proposed multi-loss function utilizes the combination of only cross-entropy loss, triplet loss, and distillation loss and gains improvement of 6.52% and 3.45% for mean average precision over ISIC-2017 and ISIC-2018. Another innovation in our methodology is a two-branch network strategy, which concurrently boosts image retrieval and classification. Through our experiments, we underscore the effectiveness and the pitfalls of diverse loss configurations in image retrieval. Furthermore, our approach underlines the advantages of retrieval-based classification through majority voting rather than relying solely on the classification head, leading to enhanced prediction for melanoma - the most lethal type of skin cancer. Our results surpass existing state-of-the-art techniques on the ISIC-2017 and ISIC-2018 datasets by improving mean average precision by 1.01% and 4.36% respectively, emphasizing the efficacy and promise of Vision Transformers paired with our tailor-made weighted loss function, especially in medical contexts. The proposed approach's effectiveness is substantiated through thorough ablation studies and an array of quantitative and qualitative outcomes. To promote reproducibility and support forthcoming research, our source code will be accessible on GitHub.

engineering, biomedical,computer science, interdisciplinary applications,mathematical & computational biology,medical informatics
Medi-CAT: Contrastive Adversarial Training for Medical Image Classification

Pervaiz Iqbal Khan,Andreas Dengel,Sheraz Ahmed

2023-11-01

Abstract:There are not many large medical image datasets available. For these datasets, too small deep learning models can't learn useful features, so they don't work well due to underfitting, and too big models tend to overfit the limited data. As a result, there is a compromise between the two issues. This paper proposes a training strategy Medi-CAT to overcome the underfitting and overfitting phenomena in medical imaging datasets. Specifically, the proposed training methodology employs large pre-trained vision transformers to overcome underfitting and adversarial and contrastive learning techniques to prevent overfitting. The proposed method is trained and evaluated on four medical image classification datasets from the MedMNIST collection. Our experimental results indicate that the proposed approach improves the accuracy up to 2% on three benchmark datasets compared to well-known approaches, whereas it increases the performance up to 4.1% over the baseline methods.

Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning
A New Perspective to Boost Vision Transformer for Medical Image Classification

Yuexiang Li,Yawen Huang,Nanjun He,Kai Ma,Yefeng Zheng

DOI: https://doi.org/10.48550/arXiv.2301.00989

2023-01-03

Computer Vision and Pattern Recognition

Abstract:Transformer has achieved impressive successes for various computer vision tasks. However, most of existing studies require to pretrain the Transformer backbone on a large-scale labeled dataset (e.g., ImageNet) for achieving satisfactory performance, which is usually unavailable for medical images. Additionally, due to the gap between medical and natural images, the improvement generated by the ImageNet pretrained weights significantly degrades while transferring the weights to medical image processing tasks. In this paper, we propose Bootstrap Own Latent of Transformer (BOLT), a self-supervised learning approach specifically for medical image classification with the Transformer backbone. Our BOLT consists of two networks, namely online and target branches, for self-supervised representation learning. Concretely, the online network is trained to predict the target network representation of the same patch embedding tokens with a different perturbation. To maximally excavate the impact of Transformer from limited medical data, we propose an auxiliary difficulty ranking task. The Transformer is enforced to identify which branch (i.e., online/target) is processing the more difficult perturbed tokens. Overall, the Transformer endeavours itself to distill the transformation-invariant features from the perturbed tokens to simultaneously achieve difficulty measurement and maintain the consistency of self-supervised representations. The proposed BOLT is evaluated on three medical image processing tasks, i.e., skin lesion classification, knee fatigue fracture grading and diabetic retinopathy grading. The experimental results validate the superiority of our BOLT for medical image classification, compared to ImageNet pretrained weights and state-of-the-art self-supervised learning approaches.
Aromatization of shikimic acid in the rat and the role of gastrointestinal micro-organisms.

D. Brewster,R. S. Jones,D. Parke

DOI: https://doi.org/10.1042/BST0040518

1976-06-01

Biochemical Society Transactions

Abstract:rats. Ether extraction of such samples at pH5.0 showed them to contain at least 98 % of the radioactivity in a conjugated form. Bile containing the labelled conjugates was diluted with 0.9% NaCl such that ~ 1 O O O d.p.s. was administered to each animal. Biliary excretion of radioactivity was again expressed as a percentage of this dose. Experimental rats received neomycin sulphate orally (100mg in 0.2ml of water) twice daily for 4 days and once on the fifth day 1-2 h before the experiment. Fig. 1 shows the observed rates of biliary excretion of mestranol after intraperitoneal and intraduodenal administration. Biliary excretion is rapid in both cases, but far more so for intraduodenal administration. Reference to the mean bile-flow rates given in Fig. 1 shows that after 3 h 35-39 % of the intraperitoneal and 72-75 % of an intraduodenal mestranol dose had been excreted in the bile. The results suggest that an active transport mechanism may be involved in the absorption of mestranol across the wall of the intestine. The biliary excretion of radioactivity after intraduodenal administration of metabolic conjugates of mestranol into normal and neomycin-treated rats is shown in Fig. 2. In addition the rate of excretion after administration of conjugated bile plus the 8-glucuronidase inhibitor saccharolactone is shown. After 4h a significant decrease in the rate of excretion of radioactivity was observed in both cases compared with controls. This lower excretion rate can be interpreted as reflecting a decrease in the rate of deconjugation, since the absorption of the free steroids and their subsequent biliary excretion is rapid (see curve for intraduodenal administration, Fig. 1). It seems on this evidence that the gut microflora are responsible (in part) for the hydrolysis of conjugated mestranol and its metabolites. The fact that recirculation is not totally prevented in neomycin-treated animals is not surprising, since neomycin does not suppress all groups of gastro-intestinal microorganisms (e.g. Bacteriodes; Finegold et al., 1965) and there is likely to be a contribution from hydrolytic enzymes of tissue origin. Preliminary incubations in vitro have shown that rat caecal micro-organisms are capable of extensive deconjugation of biliary conjugates of mestranol.
Brain Tumor Classification using Vision Transformer with Selective Cross-Attention Mechanism and Feature Calibration

Mohammad Ali Labbaf Khaniki,Alireza Golkarieh,Mohammad Manthouri

2024-06-25

Abstract:Brain tumor classification is a challenging task in medical image analysis. In this paper, we propose a novel approach to brain tumor classification using a vision transformer with a novel cross-attention mechanism. Our approach leverages the strengths of transformers in modeling long-range dependencies and multi-scale feature fusion. We introduce two new mechanisms to improve the performance of the cross-attention fusion module: Feature Calibration Mechanism (FCM) and Selective Cross-Attention (SCA). FCM calibrates the features from different branches to make them more compatible, while SCA selectively attends to the most informative features. Our experiments demonstrate that the proposed approach outperforms other state-of-the-art methods in brain tumor classification, achieving improved accuracy and efficiency. The proposed FCM and SCA mechanisms can be easily integrated into other vision transformer architectures, making them a promising direction for future research in medical image analysis. Experimental results confirm that our approach surpasses existing methods, achieving state-of-the-art performance in brain tumor classification tasks.

Image and Video Processing,Computer Vision and Pattern Recognition
TransMed: Transformers Advance Multi-Modal Medical Image Classification

Yin Dai,Yifan Gao,Fayu Liu

DOI: https://doi.org/10.3390/diagnostics11081384

IF: 3.6

2021-07-31

Diagnostics

Abstract:Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it cannot deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependencies, and effective multi-modal fusion strategies can greatly improve the performance of deep models. This prompts us to study transformer-based structures and apply them to multi-modal medical images. Existing transformer-based network architectures require large-scale datasets to achieve better performance. However, medical imaging datasets are relatively small, which makes it difficult to apply pure transformers to medical image analysis. Therefore, we propose TransMed for multi-modal medical image classification. TransMed combines the advantages of CNN and transformer to efficiently extract low-level features of images and establish long-range dependencies between modalities. We evaluated our model on two datasets, parotid gland tumors classification and knee injury classification. Combining our contributions, we achieve an improvement of 10.1% and 1.9% in average accuracy, respectively, outperforming other state-of-the-art CNN-based models. The results of the proposed method are promising and have tremendous potential to be applied to a large number of medical image analysis tasks. To our best knowledge, this is the first work to apply transformers to multi-modal medical image classification.

medicine, general & internal
HViT: Hybrid vision inspired transformer for the assessment of carotid artery plaque by addressing the cross-modality domain adaptation problem in MRI

Muhammad Ayoub,Zhifang Liao,Lifeng Li,Kelvin K L Wong

DOI: https://doi.org/10.1016/j.compmedimag.2023.102295

Abstract:Background: Medical image classification is crucial for accurate and efficient diagnosis, and deep learning frameworks have shown significant potential in this area. When a general learning deep model is directly deployed to a new dataset with heterogeneous features, the effect of domain shifts is usually ignored, which degrades the performance of deep learning models and leads to inaccurate predictions. Purpose: This study aims to propose a framework that utilized the cross-modality domain adaptation and accurately diagnose and classify MRI scans and domain knowledge into stable and vulnerable plaque categories by a modified Vision Transformer (ViT) model for the classification of MRI scans and transformer model for domain knowledge classification. Methods: This study proposes a Hybrid Vision Inspired Transformer (HViT) framework that employs a convolutional layer for image pre-processing and normalization and a 3D convolutional layer to enable ViT to classify 3D images. Our proposed HViT framework introduces a slim design with a multi-branch network and channel attention, improving patch embedding extraction and information learning. Auxiliary losses target shallow features, linking them with deeper ones, enhancing information gain, and model generalization. Furthermore, replacing the MLP Head with RNN enables better backpropagation for improved performance. Moreover, we utilized a modified transformer model with LSTM positional encoding and Golve word vector to classify domain knowledge. By using ensemble learning techniques, specifically stacking ensemble learning with hard and soft prediction, we combine the predictive power of both models to address the cross-modality domain adaptation problem and improve overall performance. Results: The proposed framework achieved an accuracy of 94.32% for carotid artery plaque classification into stable and vulnerable plaque by addressing the cross-modality domain adaptation problem and improving overall performance. Conclusion: The model was further evaluated using an independent dataset acquired from different hardware protocols. The results demonstrate that the proposed deep learning model significantly improves the generalization ability across different MRI scans acquired from different hardware protocols without requiring additional calibration data.
Towards efficient diagnostics: refining vision transformers for medical image multi-label classification

Benjamin M. Hand,Colleen P. Bailey,Aidan G. Kurz,Garrett I. Cayce

DOI: https://doi.org/10.1117/12.3013977

2024-06-07

Abstract:Medical imaging, including the use of chest x-rays, is an important tool for modern healthcare, enabling early and accurate disease diagnosis, facilitating timely interventions to mitigate health issues. By capturing images of critical internal organs like the lungs and heart, x-rays enable doctors to make informed diagnoses and treatment decisions, especially concerning respiratory and cardiac conditions. The importance of early and accurate disease diagnosis, particularly with multiple pathologies, is paramount, as it greatly impacts patient outcomes by enabling timely and specific treatments. Recently, multi-label classification has become increasingly important in medical imaging, since several pathologies can be present within a single x-ray. While traditional convolutional neural networks (CNNs) have played a pivotal role in enhancing the accuracy of x-ray diagnoses, the expanding complexity of multi-label imaging demands more sophisticated methods. Vision Transformers (ViTs) have emerged as a promising approach in medical image classification, showcasing their ability to effectively process x-ray images and identify pathologies within them. While traditional ViTs perform well, they have significant drawbacks. Most ViT models utilize a large number of parameters, often ranging from millions to billions of parameters. Such parameter-intensive designs, while powerful, are computationally heavy. This not only increases the resource requirements, but also raises concerns about their feasibility and scalability in real-world, time-sensitive healthcare settings. We propose a novel Vision Transformer architecture aimed at effectively classifying multi-label x-ray images while significantly enhancing the efficiency of ViT-based multi-label medical image classification methods. By optimizing model architectures and exploring techniques for parameter reduction, we seek to develop more streamlined and resource-efficient approaches without completely sacrificing the efficacy of these methods. Our work endeavors to bridge the gap between cutting-edge technology and practical healthcare applications, promising a more efficient and accessible future for medical image analysis.

Engineering,Medicine,Computer Science
Enhancing medical image analysis: A fusion of fully connected neural network classifier with CNN-VIT for improved retinal disease detection

Khaja Mannanuddin,V.R. Vimal,Angalkuditi Srinivas,S.D. Uma Mageswari,G. Mahendran,J. Ramya,Ashok Kumar,Pranjal Das,R.G. Vidhya

DOI: https://doi.org/10.3233/jifs-235055

2023-10-25

Journal of Intelligent & Fuzzy Systems

Abstract:Diseases of the retina continue to be a leading cause of blindness and visual impairment around the world. In the field of medical image analysis, specifically retinal disease identification, deep learning techniques, such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), have showed remarkable potential. In this paper, we present a unique method for detecting retinal diseases by combining the advantages of the Inception-V3, ResNet-50, and Vision Transformer architectures into a single model called a Cascade CNN-ViT. The suggested Cascade CNN-ViT model extracts local features from retinal pictures by leveraging the spatial hierarchy learning capabilities of Inception-V3 and ResNet-50. The Vision Transformer takes these regional characteristics and uses self-attention mechanisms to pick up global context information and long-range interdependence. The model successfully combines fine-grained local information with semantically significant global contextual cues by merging the output representations from the CNNs and Vision Transformer. undertaking comprehensive experiments on a large and varied dataset of multimodal retinal pictures to evaluate the performance of the proposed technique. Cascade CNN-ViT model outperforms standalone CNNs and Vision Transformers, as shown by the experimental findings. The model is also resilient across all classes of retinal diseases and is able to successfully deal with the complications introduced by using multiple picture types. Overall, the power of cascading Inception-V3, ResNet-50, and Vision Transformer topologies for improved retinal illness diagnosis has been demonstrated. Potentially improving the management of retinal illnesses and preserving visual health, the proposed approach could have important consequences for early detection and timely intervention.
MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentation

Gurucharan Marthi Krishna Kumar,Aman Chadha,Janine Mendola,Amir Shmuel

2024-10-04

Abstract:Large Language Models (LLMs), known for their versatility in textual data, are increasingly being explored for their potential to enhance medical image segmentation, a crucial task for accurate diagnostic imaging. This study explores enhancing Vision Transformers (ViTs) for medical image segmentation by integrating pre-trained LLM transformer blocks. Our approach, which incorporates a frozen LLM transformer block into the encoder of a ViT-based model, leads to substantial improvements in segmentation performance across various medical imaging modalities. We propose a Hybrid Attention Mechanism that combines global and local feature learning with a Multi-Scale Fusion Block for aggregating features across different scales. The enhanced model shows significant performance gains, including an average Dice score increase from 0.74 to 0.79 and improvements in accuracy, precision, and the Jaccard Index. These results demonstrate the effectiveness of LLM-based transformers in refining medical image segmentation, highlighting their potential to significantly boost model accuracy and robustness. The source code and our implementation are available at: <a class="link-external link-https" href="https://bit.ly/3zf2CVs" rel="external noopener nofollow">this https URL</a>

Image and Video Processing,Computation and Language,Computer Vision and Pattern Recognition

Multiple peritoneal calcifying fibrous tumors revealed by ischemic colitis.

Implementing vision transformer for classifying 2D biomedical images

MedViT: A robust vision transformer for generalized medical image classification

Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review

Vision Transformer for Efficient Chest X-ray and Gastrointestinal Image Classification

Vision Transformers in Medical Computer Vision -- A Contemplative Retrospection

A Novel Vision Transformer with Residual in Self-attention for Biomedical Image Classification

MetaMed: Few-shot medical image classification using gradient-based meta-learning

Using Meta-Transformers for Multimodal Clinical Decision Support and Evidence-Based Medicine

Improved EATFormer: A Vision Transformer for Medical Image Classification

Stitched vision transformer for age-related macular degeneration detection using retinal optical coherence tomography images

Echoes of images: multi-loss network for image retrieval in vision transformers

Medi-CAT: Contrastive Adversarial Training for Medical Image Classification

A New Perspective to Boost Vision Transformer for Medical Image Classification

Aromatization of shikimic acid in the rat and the role of gastrointestinal micro-organisms.

Brain Tumor Classification using Vision Transformer with Selective Cross-Attention Mechanism and Feature Calibration

TransMed: Transformers Advance Multi-Modal Medical Image Classification

HViT: Hybrid vision inspired transformer for the assessment of carotid artery plaque by addressing the cross-modality domain adaptation problem in MRI

Towards efficient diagnostics: refining vision transformers for medical image multi-label classification

Enhancing medical image analysis: A fusion of fully connected neural network classifier with CNN-VIT for improved retinal disease detection

MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentation