Abstract:Background: Echocardiograms provide vital insights into cardiac health, but their complex, multi-dimensional data presents challenges for analysis and interpretation. Current deep learning models for echocardiogram analysis often rely on supervised training, limiting their generalizability and robustness across datasets and clinical environments. Objective: To develop and evaluate EchoVisionFM (Echocardiogram video Vision Foundation Model), a self-supervised video learning framework designed to pre-train a video encoder on large-scale, unlabeled echocardiogram data. EchoVisionFM aims to produce robust and transferrable spatiotemporal representations, improving downstream performance across diverse echocardiogram datasets and clinical conditions. Methods: Our framework employs Echo-VideoMAE, an autoencoder-based video transformer that compresses and reconstructs echocardiogram video data by masking non-overlapping video patches and leveraging a ViT encoder-decoder structure. For enhanced representation, we introduce STFF-Net, a Spatio Temporal Feature Fusion Network, to integrate spatial and temporal features from the manifold representations. We pre-trained EchoVisionFM using the MIMIC-IV-ECHO dataset and fine-tuned it on the EchoNet-Dynamic dataset for downstream tasks, including classification and regression of key cardiac parameters. Results: EchoVisionFM demonstrated superior performance in classifying left ventricular ejection fraction (LVEF), achieving an accuracy of 89.12%, an F1 score of 0.9323, and an AUC of 0.9364. In regression tasks, EchoVisionFM outperformed state-of-the-art models, with LVEF prediction reaching a mean absolute error (MAE) of 4.18% and an R2 of 0.8022. The model also showed significant improvements in estimating end-systolic and end-diastolic volumes, with R2 values of 0.8006 and 0.7296, respectively. Incorporating STFF-Net led to further performance gains across tasks. Conclusion: Our results indicate that large-scale self-supervised pre-training on echocardiogram videos enables the extraction of transferable and clinically relevant features, outperforming traditional CNN-based methods. The EchoVisionFM framework, particularly with STFF-Net, enhances the extraction of spatiotemporal features, improving the predictive accuracy for various cardiac parameters. EchoVisionFM offers a powerful, scalable approach for echocardiogram analysis, with potential applications in clinical diagnostics and research.

Ultrasound Video Transformers for Cardiac Ejection Fraction Estimation

Hierarchical Vision Transformers for Cardiac Ejection Fraction Estimation

TF-Unet:An Automatic Cardiac MRI Image Segmentation Method

EchoCoTr: Estimation of the Left Ventricular Ejection Fraction from Spatiotemporal Echocardiography

Echocardiogram Vector Embeddings Via R3D Transformer for the Advancement of Automated Echocardiography

Transfer Learning Video Classification of Preserved, Mid-Range, and Reduced Left Ventricular Ejection Fraction in Echocardiography

GEMTrans: A General, Echocardiography-based, Multi-Level Transformer Framework for Cardiovascular Diagnosis

ViViEchoformer: Deep Video Regressor Predicting Ejection Fraction

Time-Distributed Vision Transformer Stacked With Transformer for Heart Failure Detection Based on Echocardiography Video

Left Ventricle Segmentation in Echocardiography with Transformer

Automatic pancreas anatomical part detection in endoscopic ultrasound videos

Echo-Vision-FM: A Pre-training and Fine-tuning Framework for Echocardiogram Videos Vision Foundation Model

Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesis

Predicting post-operative right ventricular failure using video-based deep learning

Lightweight and Interpretable Left Ventricular Ejection Fraction Estimation using Mobile U-Net

Automatic Cardiac Pathology Recognition in Echocardiography Images Using Higher Order Dynamic Mode Decomposition and a Vision Transformer for Small Datasets

A Deep Learning Based Approach for Automatic Cardiac Events Identification

Automatic segmentation of echocardiographic images using a Shifted Windows Vision Transformer architecture

Intelligent Detection of Left Ventricular Hypertrophy from Pediatric Echocardiography Videos

The Study of Echocardiography of Left Ventricle Segmentation Combining Transformer and Convolutional Neural Networks

HeartBEiT: Vision Transformer for Electrocardiogram Data Improves Diagnostic Performance at Low Sample Sizes