Abstract:This article presents a vision for the future of prosthetic devices, leveraging the advancements in large language models (LLMs) and Large Multimodal Models (LMMs) to revolutionize the interaction between humans and assistive technologies. Unlike traditional prostheses, which rely on limited and predefined commands, this approach aims to develop intelligent prostheses that understand and respond to users' needs through natural language and multimodal inputs. The realization of this vision involves developing a control system capable of understanding and translating a wide array of natural language and multimodal inputs into actionable commands for prosthetic devices. This includes the creation of models that can extract and interpret features from both textual and multimodal data, ensuring devices not only follow user commands but also respond intelligently to the environment and user intent, thus marking a significant leap forward in prosthetic technology.

What problem does this paper attempt to address?

The main issue addressed in this paper is how to utilize large language models (LLMs) and large multimodal models (LMMs) to enhance prosthetic technology for more natural and intelligent human-computer interaction. Traditional prosthetics rely on limited pre-defined commands, while this paper envisions the development of an intelligent prosthetic that can understand and respond to user's natural language and multimodal inputs. The goal is to build a control system that can translate various natural language and multimodal inputs into executable commands for the prosthetic device. This way, the prosthetic can not only follow user instructions but also intelligently respond based on the environment and user intent, significantly enhancing the effectiveness of assistive technology. The core concept of the paper is to utilize LLMs to parse and execute gait tasks and control commands for the prosthetic technology, allowing users to interact with the device through natural language and reducing the need for understanding technical details. The introduction of LMMs further enhances this interaction, combining other data sources such as images and sounds to enable the prosthetic to better understand the user's environment and walking intentions, providing more personalized adaptability. To implement this idea, training decoders are needed to map the features extracted from LLMs or LMMs to specific control parameters of the prosthetic, such as impedance, stiffness, torque, and speed. The paper also discusses how to guide LLMs and LMMs to transform high-level commands into executable low-level skills, adapting to practical rehabilitation technology applications, especially for prosthetics, orthotics, and exoskeleton control. In summary, the paper aims to address the problem of how to leverage advanced AI technologies, particularly large language and multimodal models, to innovate the control methods of prosthetic devices, improving their user-friendliness and functionality, ultimately enhancing the quality of life for people with disabilities.

Embracing Large Language and Multimodal Models for Prosthetic Technologies

Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design

LaMI: Large Language Models for Multi-Modal Human-Robot Interaction

Large Language User Interfaces: Voice Interactive User Interfaces powered by LLMs

Large Language Models Humanize Technology

Large Language Models for Robotics: A Survey

VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robots

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

Using Large Language Models to Accelerate Communication for Users with Severe Motor Impairments

When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

Adaptive Grasp Control through Multi-Modal Interactions for Assistive Prosthetic Devices

Towards Predictive Communication with Brain-Computer Interfaces integrating Large Language Models

Probing Language Models' Gesture Understanding for Enhanced Human-AI Interaction

Large language models in physical therapy: time to adapt and adept

Empowering Large Language Models on Robotic Manipulation with Affordance Prompting

How Can Large Language Models Help Humans in Design and Manufacturing?

Integrating large language models in care, research, and education in multiple sclerosis management

Deep Learning for Enhanced Prosthetic Control: Real-Time Motor Intent Decoding for Simultaneous Control of Artificial Limbs

Broadband Prosthetic Interfaces: Combining Nerve Transfers and Implantable Multichannel EMG Technology to Decode Spinal Motor Neuron Activity

ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation