A CNN and LSTM Network for Eye-Blink Classification from MRI Scanner Monitoring Videos

Ronan Bennett,Shantanu H Joshi

DOI: https://doi.org/10.1109/EMBC46164.2021.9629937

Abstract:Eye closure changes brain activity, so eye-blink tracking of subjects undergoing resting-state functional magnetic resonance imaging (fMRI) is relevant for identifying when a subject blinks, falls asleep, or keeps their eyes closed. Existing MRI eye-tracking solutions use commercially available MR-compatible video cameras with tracking software that can fail on low-quality videos. In this paper, we propose a two-stage convolutional recurrent neural network to classify open and closed eyes from frames of MRI eye-tracking videos under variable camera conditions. The model extracts visual features from each video frame using a convolutional neural network based on the Inception-v3 model, then uses a long short-term memory network to incorporate temporal information encoded in the sequence of visual features over time. Our model is implemented in Keras and demonstrated on a dataset of MRI eye-tracking videos from the Human Connectome Project. We manually labelled frames from the dataset for training and evaluation. The network was able to classify eye-blink states with a precision of 0.739 and recall of 0.835 on a previously unseen holdout dataset under varying camera conditions, eye position, and video quality.Clinical relevance- Functional mapping studies in psychiatry and neuro-development which rely on a resting state fMRI protocol may yield divergent results depending on whether the subject keeps their eyes closed or open or whether the subject falls asleep. The clinical relevance of this work is to introduce the eye state (closed or open) in brain imaging studies as a prospective covariate, and as a feature that can potentially control for sleep state as a confounding factor.