Abstract:The recent advances in Multimodal AI & Generative AI open doors to the possibilities of solving key challenges for Persons with Learning Disability. To assist individuals facing difficulty in visual or auditory perception, this paper designs & develops a multimodal AI agent using recent advances in the field. We aim to solve the challenge of enabling persons with Visual or Auditory Processing Disorders to learn & communicate. We do this by exploring a design that allows the transformation of information across visual and language modalities. This design can be realized with the recent advances in Generative Multimodal AI. Based on each individual's needs, the AI agent dynamically adapts the Human Computer interaction model. For instance, for a child with Visual Processing Disorder (VPD), given the child's hindered ability to make sense of information taken in through the eyes, the Multimodal AI agent transforms any visual information into auditory user interaction. In another instance, for a person with Central Auditory Processing Disorder (CAPD), given the hindrance in the individual's ability to analyze information taken in through the ears, the AI dynamically translates any speech modality into visual cues. Thus the AI agent adapts dynamically to the strengths and abilities of the individual. To enable students with VPD to learn, the design allows the student to ask questions about an image. This design is realized as a Visual Question Answering task in Vision Language Transformer models. We explore interactive multimodal conversations with Few shot Learning and In-Context Instruction Tuning of Multimodal Large Language Models to address difficulty in visual reasoning. To enable persons with CAPD to learn, the design translates audio lectures into visual cues. This visual cue consists of a combination of words using speech recognition and Large Language Models based re-phrasing to simpler words, cross-modal retrieval of images to address auditory memory challenges, and AI-generated images. To identify the strengths of each child, we also explore Multimodal embedding based Multimodal latent space arithmetic to link AI across senses. To effectively integrate the proposed design into the mainstream, we explore a universal design based inclusive approach to extend the use case to create AI assistants for assisting children with different learning styles such as visual learners or auditory learners. To enable future research on the proposed design, we explore an architecture to compose a pipeline of AI models, and to connect with external systems via plugin connectors. We implement lab scale prototypes of this design and present a demo on the project webpage at https://sites.google.com/view/multimodallearningdisability.

Motion Design Principles for Accessible Video-based Learning: Addressing Cognitive Challenges for Deaf and Hard of Hearing Learners

Right mix of speech and non-speech: hybrid auditory feedback in mobility assistance of the visually impaired

An Eye Tracking Study: Positive Emotional Interface Design Facilitates Learning Outcomes in Multimedia Learning?

Inclusive Emotion Technologies: Addressing the Needs of d/Deaf and Hard of Hearing Learners in Video-Based Learning

Video accessibility enhancement for hearing-impaired users

Teaching Video Game Design Accessibility: Toward Effective Pedagogic Interventions in Accessible Design

Four-Component Instructional Design Model(I):Multimedia Principles in Environments for complex Learning

How Technology Should Be Committed to the Promotion of Learning:Mayer's View on Multimedia Learning and Teaching Designing

"Real Learner Data Matters" Exploring the Design of LLM-Powered Question Generation for Deaf and Hard of Hearing Learners

The Four-Component Instructional Design Model: Multimedia Principles in Environments for Complex Learning

"It's Kind of Context Dependent": Understanding Blind and Low Vision People's Video Accessibility Preferences Across Viewing Scenarios

Research on immersive interaction design based on visual and tactile feature analysis of visually impaired children

Towards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing

Multi-sensory learning framework for visually impaired learners: Use of 3D, haptic, audio, olfactory media

Caring for Special Participants in the Digital Media Era: A Study on Enhancing the Blind User Experience on Short Video Platforms Through Auditory Cues

Application of User Experience and Design Thinking to the Construction of a Class Assistance System for Hearing- and Speech-Impaired People

Designing A Distance Learning System for English Reading by Applying Multimodal Discourse Analysis Theory

Gamification Design Based Research on Speech Training System for Hearing-Impaired Children

Remotely Co-Designing Features for Communication Applications using Automatic Captioning with Deaf and Hearing Pairs

Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing

Design of Generative Multimodal AI Agents to Enable Persons with Learning Disability