HandMol: Coupling WebXR, AI and HCI technologies for Immersive, Natural, Collaborative and Inclusive Molecular Modeling

Luciano Abriata,Fabio Cortes Rodriguez,Lucien Krapp,Matteo Dal Peraro

DOI: https://doi.org/10.26434/chemrxiv-2023-5bj9j

2023-11-27

Abstract:Except for isolated developments and specific software extensions, molecular graphics and modeling have historically been stuck at flat screens for visualization, mouse operations for molecular manipulation, menus and command line interfaces for controls, and single-user interfaces that only allow collaboration by streaming video hence limited to just sharing the view of the user operating the software. We demonstrate here how various technologies are ripe enough to enable much more fluent, immersive and natural human-computer interactions that in turn facilitate collaboration between human users, using affordable hardware through the internet and without even installing any specialized programs. For this, we introduce HandMol, a web app that exploits (i) WebXR for molecular visualization and manipulation in virtual reality, (ii) speech recognition coupled to a large language model to pass commands orally, (iii) speech synthesis for auditory feedback, (iv) WebRTC to communicate multiple instances of the tool without even requiring a server, and (v) external APIs to flexibly account for molecular mechanics, exemplified here with an endpoint running an AMBER forcefield for protein and nucleic acids and another running a DFT-trained neural network, ANI-2x, to allow exploration of conformation and some simple reactivity at high speed and accuracy. We show example applications to situations from daily work and education in chemistry and structural biology where HandMol can provide an advantage over traditional software: exploring and explaining molecular conformations and reactivity, docking and undocking small molecules into/out of protein pockets, threading molecules through nanopores, preparing systems for molecular simulations and for protein design, etc. We also present a brief study showing how users, even with limited or even no experience in VR, can significantly benefit from these kinds of technologies. As a draft prototype for the moment, HandMol is made available free of charge and without registration at https://go.epfl.ch/handmol, in (optional but greatly appreciated) exchange for feedback on usability and on features expected for this kind of tools.

Chemistry

What problem does this paper attempt to address?

The paper attempts to address the limitations of traditional molecular graphics and modeling software in terms of visualization, manipulation, and collaboration. Specifically:

1. **Visualization Limitations**: Traditional molecular graphics software mainly relies on 2D screen displays for 3D structures, lacking depth perception and making it difficult to intuitively present the 3D characteristics of molecules.
2. **Manipulation Limitations**: Traditional molecular manipulation is primarily done through a mouse, which is not natural enough and requires users to master complex commands and shortcuts.
3. **Collaboration Limitations**: Traditional software interfaces usually allow only a single user to operate, while other users can only watch passively, unable to interact and collaborate in real-time.

To address these issues, the authors introduce HandMol, a web application that combines WebXR, Artificial Intelligence (AI), and Human-Computer Interaction (HCI) technologies. The main features of HandMol include:

- **Virtual Reality (VR)**: Using WebXR technology, users can visualize and manipulate molecules in virtual reality, providing a more immersive experience.
- **Voice Recognition and Synthesis**: Through voice recognition technology, users can issue commands verbally, simplifying the operation process; at the same time, voice synthesis provides auditory feedback to users.
- **Multi-Device Support**: HandMol can run on various devices, including VR headsets, smartphones, tablets, and computers, without the need to install any specialized programs.
- **Real-Time Collaboration**: Using WebRTC technology, multiple users can collaborate in real-time over the internet without server support.
- **Molecular Mechanics**: By calling external APIs, HandMol can flexibly handle molecular mechanics calculations, such as using the AMBER force field or the ANI-2x neural network for high-precision energy and force calculations.

Through the integration of these technologies, HandMol aims to provide a smoother, more natural, collaborative, and inclusive molecular modeling tool suitable for research and educational scenarios in chemistry and structural biology.