TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types

Wolfgang Fuhl,Gjergji Kasneci,Enkelejda Kasneci

DOI: https://doi.org/10.1109/ismar52148.2021.00053

2023-06-06

Abstract:We present TEyeD, the world's largest unified public data set of eye images taken with head-mounted devices. TEyeD was acquired with seven different head-mounted eye trackers. Among them, two eye trackers were integrated into virtual reality (VR) or augmented reality (AR) devices. The images in TEyeD were obtained from various tasks, including car rides, simulator rides, outdoor sports activities, and daily indoor activities. The data set includes 2D and 3D landmarks, semantic segmentation, 3D eyeball annotation and the gaze vector and eye movement types for all images. Landmarks and semantic segmentation are provided for the pupil, iris and eyelids. Video lengths vary from a few minutes to several hours. With more than 20 million carefully annotated images, TEyeD provides a unique, coherent resource and a valuable foundation for advancing research in the field of computer vision, eye tracking and gaze estimation in modern VR and AR applications. Download: Just connect via FTP as user TEyeDUser and without password to <a class="link-external link-http" href="http://nephrit.cs.uni-tuebingen.de" rel="external noopener nofollow">this http URL</a> (<a class="link-external link-ftp" href="ftp://nephrit.cs.uni-tuebingen.de" rel="external noopener nofollow">this ftp URL</a>).

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the need for high-quality eye movement datasets in computer vision, eye tracking, and modern virtual reality (VR) and augmented reality (AR) applications. Specifically, the paper introduces the **TüEyeD** dataset, which is currently the largest publicly available eye movement image dataset in the world, containing over 20 million real-world eye movement images. These images were collected using seven different head-mounted eye tracking devices and cover a variety of tasks and activities, such as driving, simulator operations, indoor and outdoor activities, etc.

### Main Contributions

1. **Large-scale, unified, and richly annotated dataset**: TüEyeD provides over 20 million eye movement images, which are annotated with detailed 2D and 3D segmentation, 2D and 3D landmarks, eyeball models, gaze vectors, and eye movement types (such as fixations and saccades).
2. **Extensive coverage of tasks and activities**: The dataset includes data from various scenarios, ranging from driving and simulator operations to indoor and outdoor activities.
3. **Detailed annotations**: It provides detailed annotations of 2D and 3D segmentation, 2D and 3D landmarks, pupil centers, gaze vectors, eyeball models, and eye movement types.
4. **Real-world challenges**: The dataset includes various real-world challenges, such as different resolutions, steep viewing angles, changing lighting conditions, and device slippage.

### Application Prospects

The extensive and high-quality nature of the TüEyeD dataset makes it highly valuable for multiple fields:

- **Computer vision and machine learning**: It can serve as an annotated dataset for developing tasks such as eye segmentation, iris recognition, and feature extraction.
- **Eye tracking research**: It helps improve the quality of eye position and gaze estimation models, as well as the automatic classification of eye movement types.
- **Cognitive science**: It can be used to develop models for assessing user cognitive states (such as cognitive load or fatigue).
- **Virtual reality and augmented reality**: Eye tracking is considered a key enabling technology for these fields, as it can save computational and rendering resources and provide a more intuitive and immersive user experience.

### Conclusion

The TüEyeD dataset fills the gap in existing datasets by providing a large amount of high-quality, diverse annotated data, thus laying a solid foundation for research and applications in related fields.