Apple Vision Pro: what does it mean for scientists?

Jonathan O'Callaghan

DOI: https://doi.org/10.1038/d41586-024-00387-z

IF: 64.8

2024-02-13

Nature

Abstract:The headset opens up possibilities in accessibility and medical research — and raises concerns about human behaviour. The headset opens up possibilities in accessibility and medical research — and raises concerns about human behaviour.

multidisciplinary sciences

What problem does this paper attempt to address?

Apple Vision Pro for Ophthalmology and Medicine

Mouayad Masalkhi,Ethan Waisberg,Joshua Ong,Nasif Zaman,Prithul Sarker,Andrew G. Lee,Alireza Tavakkoli

DOI: https://doi.org/10.1007/s10439-023-03283-1

IF: 3.8

2023-06-18

Annals of Biomedical Engineering

Abstract:The emergence of new technologies continues to break barriers and transform the way we perceive and interact with the world. In this scientific article, we explore the potential impact of the new Apple XR headset on revolutionizing accessibility for individuals with visual deficits. With its rumored exceptional 4-K displays per eye and 5000 nits of brightness, this headset has the potential to enhance the visual experience and provide a new level of accessibility for users with visual impairments. We delve into the technical specifications, discuss the implications for accessibility, and envision how this groundbreaking technology could open up new possibilities for individuals with visual deficits.

engineering, biomedical
Apple Vision Pro: Comments in Healthcare

Ezequiel Santos,Vanessa Castillo

2024-04-02

Abstract:This paper objectively analyzes the emerging discourse surrounding Apple Vision Pro's application in healthcare and medical education. Released in June 2023, Apple Vision Pro represents a significant advancement in spatial computing, combining augmented and virtual reality to create new possibilities in digital interaction. We aim to compile and present recent articles. We used PubMed, IEEE Xplore, Google Scholar, and JSTOR. Non-academic publications were excluded. The results were six commentaries, one a pre-print. All were majorly optimistic, with one mentioning VR/AR sickness. For future research directions, we stress the need for continued exploration of Apple Vision Pro's capabilities and limitations and expect expert opinions to englobe this discussion.

Human-Computer Interaction
Is the Apple Vision Pro the Ultimate Display? A First Perspective and Survey on Entering the Wonderland of Precision Medicine

Jan Egger,Christina Gsaxner,Gijs Luijten,Jianxu Chen,Xiaojun Chen,Jiang Bian,Jens Kleesiek,Behrus Puladi

DOI: https://doi.org/10.2196/52785

2024-09-18

JMIR Serious Games

Abstract:At the Worldwide Developers Conference in June 2023, Apple introduced the Vision Pro. The Apple Vision Pro (AVP) is a mixed reality headset; more specifically, it is a virtual reality device with an additional video see-through capability. The video see-through capability turns the AVP into an augmented reality (AR) device. The AR feature is enabled by streaming the real world via cameras on the (virtual reality) screens in front of the user's eyes. This is, of course, not unique and is similar to other devices, such as the Varjo XR-3 (Varjo Technologies Oy). Nevertheless, the AVP has some interesting features, such as an inside-out screen that can show the headset wearer's eyes to "outsiders," and a button on the top, called the "digital crown," that allows a seamless blend of digital content with the user's physical space by turning it. In addition, it is untethered, except for the cable to the battery, which makes the headset more agile, compared to the Varjo XR-3. This could actually come closer to "The Ultimate Display," which Ivan Sutherland had already sketched in 1965. After a great response from the media and social networks to the release, we were able to test and review the new AVP ourselves in March 2024. Including an expert survey with 13 of our colleagues after testing the AVP in our institute, this Viewpoint explores whether the AVP can overcome clinical challenges that AR especially still faces in the medical domain; we also go beyond this and discuss whether the AVP could support clinicians in essential tasks to allow them to spend more time with their patients.
Apple Vision Pro and why extended reality will revolutionize the future of medicine

Ethan Waisberg,Joshua Ong,Mouayad Masalkhi,Nasif Zaman,Prithul Sarker,Andrew G. Lee,Alireza Tavakkoli

DOI: https://doi.org/10.1007/s11845-023-03437-z

2023-06-28

Abstract:Apple unveiled its highly anticipated mixed-reality headset, called the Apple Vision Pro on June 5, 2023. The primary user interface relies on eye tracking, hand, gestures, cameras, and sensors, eliminating the need for physical controllers such as keyboards or touch screens. The refined capabilities of this technology can be utilized for diverse purposes, including but not limited to medical and surgical education, and remote medical consultations. All things considered, virtual reality is a highly promising area for the future of medicine, from improving medical education and vision screening to physical and psychological rehabilitation. We look forward to further innovations in this exciting area for years to come.
Apple Vision Pro for Healthcare: "The Ultimate Display"? -- Entering the Wonderland of Precision Medicine

Jan Egger,Christina Gsaxner,Xiaojun Chen,Jiang Bian,Jens Kleesiek,Behrus Puladi

2023-10-10

Abstract:At the Worldwide Developers Conference (WWDC) in June 2023, Apple introduced the Vision Pro. The Vision Pro is a Mixed Reality (MR) headset, more specifically it is a Virtual Reality (VR) device with an additional Video See-Through (VST) capability. The VST capability turns the Vision Pro also into an Augmented Reality (AR) device. The AR feature is enabled by streaming the real world via cameras to the (VR) screens in front of the user's eyes. This is of course not unique and similar to other devices, like the Varjo XR-3. Nevertheless, the Vision Pro has some interesting features, like an inside-out screen that can show the headset wearers' eyes to "outsiders" or a button on the top, called "Digital Crown", that allows you to seamlessly blend digital content with your physical space by turning it. In addition, it is untethered, except for the cable to the battery, which makes the headset more agile, compared to the Varjo XR-3. This could actually come closer to the "Ultimate Display", which Ivan Sutherland had already sketched in 1965. Not available to the public yet, like the Ultimate Display, we want to take a look into the crystal ball in this perspective to see if it can overcome some clinical challenges that - especially - AR still faces in the medical domain, but also go beyond and discuss if the Vision Pro could support clinicians in essential tasks to spend more time with their patients.

Artificial Intelligence,Graphics,Human-Computer Interaction
Apple Vision Pro: A Paradigm Shift in Medical Technology

Nandan M Shanbhag,Abdulrahman Bin Sumaida,Khalifa Al Shamisi,Khalid Balaraj

DOI: https://doi.org/10.7759/cureus.69608

2024-09-17

Cureus

Abstract:The introduction of Apple Vision Pro (AVP) marks a significant milestone in the intersection of technology and healthcare, offering unique capabilities in mixed reality, which Apple terms "spatial computing." This narrative review aims to explore the various applications of AVP in medical technology, emphasizing its impact on patient care, clinical practices, medical education, and future directions. The review synthesizes findings from multiple studies and articles published between January 2023 and May 2024, highlighting AVP's potential to enhance visualization in diagnostic imaging and surgical planning, assist visually impaired patients, and revolutionize medical education through immersive learning environments. Despite its promise, challenges remain in integrating AVP into existing healthcare systems and understanding its long-term impact on patient outcomes. As research continues, AVP is poised to play a pivotal role in the future of medicine, offering a transformative tool for healthcare professionals.
The future of ophthalmology and vision science with the Apple Vision Pro

Ethan Waisberg,Joshua Ong,Mouayad Masalkhi,Nasif Zaman,Prithul Sarker,Andrew G. Lee,Alireza Tavakkoli

DOI: https://doi.org/10.1038/s41433-023-02688-5

IF: 4.4563

2023-08-05

Eye

Abstract:Eye, Published online: 04 August 2023; doi:10.1038/s41433-023-02688-5 The future of ophthalmology and vision science with the Apple Vision Pro

ophthalmology
The Apple Vision Pro as a Neurosurgical Planning Tool: A Case Report

Joshua Olexa,Annie Trang,Jonathan Cohen,Kevin Kim,Maureen Rakovec,Jordan Saadon,Charles Sansur,Graeme Woodworth,Gary Schwartzbauer,Jacob Cherian

DOI: https://doi.org/10.7759/cureus.54205

2024-02-14

Cureus

Abstract:With its recent release, the Apple Vision Pro (Apple Inc., Cupertino, CA) represents a promising technological advancement of mixed reality in the field of neurosurgery and medicine more broadly. With all new technologies, it is critical to facilitate early use and assessment of the technology to facilitate adoption by the larger medical community. A 44-year-old female with a history of ruptured intracranial aneurysm status post anterior communicating artery aneurysm clipping presented with worsened confusion and intermittent headache. CT imaging revealed evidence of hydrocephalus due to the malfunction of a previous right parietal ventriculoperitoneal (VP) shunt. Prior to the case, the Apple Vision Pro was used in the operating room to visualize and interact with a 3D model of the patient's anatomy for the patient undergoing a VP shunt placement. A visualization of the 3D model through the headset was used to plan the approach and entry point. At the conclusion of the procedure, all clinicians and operating staff who used the technology for planning completed a survey about their initial impressions of the headset. Overall, users felt the 3D models felt realistic (4.5/5), that the display of the user's real-world view felt natural (4.3/5), and that the headset did not cause eye strain or fatigue (4.5/5). The majority of users responded that they would continue to use the headset for cases (4/5). This represents one of the first known clinical uses of the Apple Vision Pro. It is a cutting-edge technology that will likely provide immense value for healthcare providers as it becomes more integrated into clinical care.
Apple Vision Pro 在元宇宙医学中的应用

DOI: https://doi.org/10.61189/769612hdtofy

2024-03-28

Abstract:Apple Vision Pro利用虚拟现实和增强现实技术,能够为医学教育、临床诊断与治疗以及医疗管理等领域带来革命性的变革和贡献,推动元宇宙医学的发展和应用。然而,实际应用Apple Vision Pro的过程中也需要解决技术和伦理挑战,包括数据隐私、安全性问题、医疗责任和法律法规考虑等。本文旨在探索Apple Vision Pro在元宇宙医学场景中的应用,以期推广Apple Vision Pro在医学领域的应用,促进医学教育、临床实践和医疗管理的全面进步。
Interpreting Apple’s visions: Examining the spatiality of the Apple Vision Pro

Tyler Blackman,Daniel Harley

DOI: https://doi.org/10.1177/29768624241283913

2024-01-01

Abstract:This paper extends previous empirical and theoretical research by applying a critical analysis to the recent discourse of “spatial computing,” a term used by Apple Inc. in the launch of the Apple Vision Pro headset. We focus primarily on advertisements and other official videos used to promote and communicate the value(s) of Apple's inaugural “spatial computer,” and how this spatiality is mediated through the hardware and software of the Apple Vision Pro. With techno-utopian visions of daily life that belie Apple's broader platform politics, we argue that Apple's promotional content reveals sociocultural, embodied, material, spatial, and other imaginaries of extended reality that present Apple's novel mediations and data capture as normal and desirable for users who are framed as affluent, able-bodied tech enthusiasts participating in a digital economy that blurs work and leisure.
Ethical challenges of technological advance: Apple Vision Pro

Efraim Caprioli,Jonathan de Barros Vitta,Henrique Lacerda Nieddermeyer,Paulo Pardo

DOI: https://doi.org/10.54033/cadpedv21n2-033

2024-02-14

Abstract:Addressing the ethical developments of the technology involved in the Apple Vision Pro device that proposes access to virtual and augmented reality, we address the evolution of the technology involved in contrast to the slow development of law, highlighting the need for regulation adaptable to social and technological changes. This article aims to propose a discussion about the role of law in user privacy considering legal and ethical implications, especially in relation to the rights of privacy and self-determination. In this context, the influence and neutrality of algorithms are addressed from the aspect of political control, raising the need for a dialogue between science, technology and their influence on democracy so that this technology does not interfere with the protection of individual and collective rights. The present study is based on exploratory research, with bibliographic analysis based on data collection on updated scientific material on the topic.
A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

Yaozu Ye,Kaiwei Wang,Weijian Hu,Huabing Li,Kailun Yang,Lei Sun,Zuobing Chen

DOI: https://doi.org/10.1088/1742-6596/1229/1/012026

2019-01-01

Abstract:There are a very few people who have the ability to "see" the surroundings by the echoes, which is called echolocation. The study of the brain mechanism of echolocation can not only help to improve the blind assistance device, but also provides a window into the research of brain's plasticity. In this paper, we developed a wearable system to transform the spatial information captured by camera into a voice description and fed it back to blind users which is inspired by echolocation. After our online virtual scene training, users can easily discriminate object location in the camera's view, motion of the objects, even shape of the objects. Compared with natural echolocation, it's easier to learn and be applied in daily life. In addition, the device achieves high spacial resolution. In this study, two trained blind subjects and two non-trained sighted subjects were tested by using functional Magnetic Resonance Imaging (fMRI). We obtain the fMRI images of the subjects' brain activity when they were listening to the sound of the wearable prototype. Intriguingly, we find that after training with the blind assistance system, the blind' visual area of the brain have been activated when they are dealing with the acoustic feedback from the device.
Digital Eyes: Social Implications of XR EyeSight

Maurizio Vergari,Tanja Kojić,Wafaa Wardah,Maximilian Warsinke,Sebastian Möller,Jan-Niklas Voigt-Antons,Robert P. Spang

DOI: https://doi.org/10.1145/3641825.3689526

2024-10-03

Abstract:The EyeSight feature, introduced with the new Apple Vision Pro XR headset, promises to revolutionize user interaction by simulating real human eye expressions on a digital display. This feature could enhance XR devices' social acceptability and social presence when communicating with others outside the XR experience. In this pilot study, we explore the implications of the EyeSight feature by examining social acceptability, social presence, emotional responses, and technology acceptance. Eight participants engaged in conversational tasks in three conditions to contrast experiencing the Apple Vision Pro with EyeSight, the Meta Quest 3 as a reference XR headset, and a face-to-face setting. Our preliminary findings indicate that while the EyeSight feature improves perceptions of social presence and acceptability compared to the reference headsets, it does not match the social connectivity of direct human interactions.

Human-Computer Interaction
Apple Intelligence in neurosurgery

Ali A Mohamed,Brandon Lucke-Wold

DOI: https://doi.org/10.1007/s10143-024-02568-w

2024-07-15

Abstract:With the current artificial intelligence (AI) boom, new innovative and accessible applications requiring minimal computer science expertise have been developed for discipline specific and mainstream purposes. Apple Intelligence, a new AI model developed by Apple, aims to enhance user experiences with new functionalities across many of its product offerings. Although designed for the everyday user, many of these advances have potential applications in neurosurgery. These include functionalities for writing, image generation, and upgraded integrations to the voice command assistant Siri. Future integrations may also include other Apple products such as the vision pro for preoperative and intraoperative applications. Considering the popularity of Apple products, particularly the iPhone, it is important to appraise this new technology and how it can be leveraged to enhance patient care, improve neurosurgical education, and facilitate more efficiency for the neurosurgeon.
Measuring eye-tracking accuracy and its impact on usability in apple vision pro

Zehao Huang,Gancheng Zhu,Xiaoting Duan,Rong Wang,Yongkai Li,Shuai Zhang,Zhiguo Wang

2024-08-14

Abstract:With built-in eye-tracking cameras, the recently released Apple Vision Pro (AVP) mixed reality (MR) headset features gaze-based interaction, eye image rendering on external screens, and iris recognition for device unlocking. One of the technological advancements of the AVP is its heavy reliance on gaze- and gesture-based interaction. However, limited information is available regarding the technological specifications of the eye-tracking capability of the AVP, and raw gaze data is inaccessible to developers. This study evaluates the eye-tracking accuracy of the AVP with two sets of tests spanning both MR and virtual reality (VR) applications. This study also examines how eye-tracking accuracy relates to user-reported usability. The results revealed an overall eye-tracking accuracy of 1.11° and 0.93° in two testing setups, within a field of view (FOV) of approximately 34° x 18°. The usability and learnability scores of the AVP, measured using the standard System Usability Scale (SUS), were 75.24 and 68.26, respectively. Importantly, no statistically reliable correlation was found between eye-tracking accuracy and usability scores. These results suggest that eye-tracking accuracy is critical for gaze-based interaction, but it is not the sole determinant of user experience in VR/AR.

Human-Computer Interaction
¿Como pueden aportar las Apple Vision Pro en cirugía plástica?

Diego Sanchez-Martínez,Julián Padilla-Rojas,William Diaz-Chaker

DOI: https://doi.org/10.6018/edumed.606141

2024-03-11

Revista Española de Educación Médica

Abstract:Technological advances have marked a milestone in traditional educational methods. The Apple Vision Pro comes with immersive technology of the highest quality, leading us in a direction in which residents and specialists can interact in a safe space. These glasses allow to improve surgical techniques, carefully study the anatomy of patients, recreate complex clinical cases, make video calls with other colleagues in real time to clarify doubts and also guide diagnoses, procedures and treatments, among other functions. Making this device an academic tool with unlimited potential for training in plastic surgery and other areas of medicine. Los avances tecnológicos han marcado un hito en los métodos de educación tradicionales. Las Apple Vision Pro trae consigo una tecnología inmersiva de la mejor calidad, que nos encamina en una dirección en la que residentes y especialistas pueden interactuar en un espacio seguro. Estas gafas permiten mejorar técnicas las quirúrgicas, estudiar minuciosamente la anatomía de los pacientes, recrear casos clínicos complejos, realizar videollamadas con otros colegas en tiempo real para aclarar dudas y también orientar diagnósticos, procedimientos y tratamientos, entre otras funciones. Convirtiendo este dispositivo una herramienta académica con un potencial sin límites para el entrenamiento en cirugía plástica y otras áreas de la medicina.
The future of brain–computer interfaces in medicine

Paul Webster

DOI: https://doi.org/10.1038/d41591-024-00031-3

IF: 82.9

2024-05-08

Nature Medicine

Abstract:Growing interest in non-invasive brain–computer interfaces, rather than implants, might improve accessibility for patients, but resolution needs to be improved. Growing interest in non-invasive brain–computer interfaces, rather than implants, might improve accessibility for patients, but resolution needs to be improved.

biochemistry & molecular biology,cell biology,medicine, research & experimental
Eyes: The Gateway to Brain Health—Advancing Disease Detection and Patient Accessibility

Sai Mattapalli,Rohan Kalahasty

DOI: https://doi.org/10.30953/thmt.v8.438

2023-11-03

Telehealth and Medicine Today

Abstract:In the past five years, artificial intelligence has revolutionized healthcare by enabling earlier detection, diagnosis, and treatment of diseases. A recent breakthrough involves using eye-tracking technology to identify ocular biomarkers, aiding in the early detection of conditions like Alzheimer's and neurodegenerative diseases. Eye movement tests, previously costly and complex, have become accessible through smartphone and laptop cameras. These tests help detect a range of conditions, from degenerative disorders to brain injuries, improving patient outcomes and facilitating telehealth. This technology not only benefits patients but also advances the study of brain health. By delving into neurology and treatment efficacy, it may provide insights into the causes of neurodegenerative diseases. The future potential of gaze tracking is promising, with non-invasive, cost-effective systems offering hope for earlier diagnosis and treatment of neurodegenerative diseases, addressing a significant gap in healthcare. The eyes are not just windows to the soul but also a gateway to the brain's health.
Brain Recording, Mind-Reading, and Neurotechnology: Ethical Issues from Consumer Devices to Brain-Based Speech Decoding

Stephen Rainey,Stéphanie Martin,Andy Christen,Pierre Mégevand,Eric Fourneret

DOI: https://doi.org/10.1007/s11948-020-00218-0

IF: 3.777

2020-04-30

Science and Engineering Ethics

Abstract:Abstract Brain reading technologies are rapidly being developed in a number of neuroscience fields. These technologies can record, process, and decode neural signals. This has been described as ‘mind reading technology’ in some instances, especially in popular media. Should the public at large, be concerned about this kind of technology? Can it really read minds? Concerns about mind-reading might include the thought that, in having one’s mind open to view, the possibility for free deliberation, and for self-conception, are eroded where one isn’t at liberty to privately mull things over. Themes including privacy, cognitive liberty, and self-conception and expression appear to be areas of vital ethical concern. Overall, this article explores whether brain reading technologies are really mind reading technologies. If they are, ethical ways to deal with them must be developed. If they are not, researchers and technology developers need to find ways to describe them more accurately, in order to dispel unwarranted concerns and address appropriately those that are warranted.

ethics,history & philosophy of science,multidisciplinary sciences,engineering, multidisciplinary
A First Look at Immersive Telepresence on Apple Vision Pro

Ruizhi Cheng,Nan Wu,Matteo Varvello,Eugene Chai,Songqing Chen,Bo Han

DOI: https://doi.org/10.1145/3646547.3689006

2024-09-12

Abstract:Due to the widespread adoption of "work-from-home" policies, videoconferencing applications (e.g., Zoom) have become indispensable for remote communication. However, they often lack immersiveness, leading to the so-called "Zoom fatigue" and degrading communication efficiency. The recent debut of Apple Vision Pro, a mobile headset that supports "spatial persona", aims to offer an immersive telepresence experience. In this paper, we conduct a first-of-its-kind in-depth and empirical study to analyze the performance of immersive telepresence with Apple FaceTime, Cisco Webex, Microsoft Teams, and Zoom on Vision Pro. We find that only FaceTime provides a truly immersive experience with spatial personas, whereas others still operate 2D personas. Our measurement results reveal that (1) FaceTime delivers semantic data to optimize bandwidth consumption, which is even lower than that of 2D persona for other applications, and (2) it employs visibility-aware optimizations to reduce rendering overhead. However, the scalability of FaceTime remains limited, with a simple server-allocation strategy that potentially leads to high network delay for users.

Networking and Internet Architecture

Apple Vision Pro: what does it mean for scientists?

Apple Vision Pro for Ophthalmology and Medicine

Apple Vision Pro: Comments in Healthcare

Is the Apple Vision Pro the Ultimate Display? A First Perspective and Survey on Entering the Wonderland of Precision Medicine

Apple Vision Pro and why extended reality will revolutionize the future of medicine

Apple Vision Pro for Healthcare: "The Ultimate Display"? -- Entering the Wonderland of Precision Medicine

Apple Vision Pro: A Paradigm Shift in Medical Technology

The future of ophthalmology and vision science with the Apple Vision Pro

The Apple Vision Pro as a Neurosurgical Planning Tool: A Case Report

Apple Vision Pro 在元宇宙医学中的应用

Interpreting Apple’s visions: Examining the spatiality of the Apple Vision Pro

Ethical challenges of technological advance: Apple Vision Pro

A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

Digital Eyes: Social Implications of XR EyeSight

Apple Intelligence in neurosurgery

Measuring eye-tracking accuracy and its impact on usability in apple vision pro

¿Como pueden aportar las Apple Vision Pro en cirugía plástica?

The future of brain–computer interfaces in medicine

Eyes: The Gateway to Brain Health—Advancing Disease Detection and Patient Accessibility

Brain Recording, Mind-Reading, and Neurotechnology: Ethical Issues from Consumer Devices to Brain-Based Speech Decoding

A First Look at Immersive Telepresence on Apple Vision Pro