Abstract:Background: Aphasia, resulting from acquired brain injury, disrupts language processing and usage, significantly impacting individuals' social communication and life participation. Given the limitations of traditional assessments in capturing the nuanced challenges faced by individuals with aphasia, this study seeks to explore the potential benefits of integrating multimodal communication elements into discourse analysis to better capture narrative proficiency in this population. Objective: This study examined how incorporating multimodal communication elements (e.g., physical gestures, writing, drawing) into discourse analysis may affect the narrative outcomes of persons with aphasia compared to those observed using methods that exclude multimodal considerations. Methods: Participants included individuals with chronic aphasia and age-and education-matched healthy controls who completed a storytelling task—the Bear and the Fly story. Macrolinguistic scores were obtained using verbal-only and multimodal scoring approaches. Additionally, the frequency and type of multimodal communication use during storytelling were examined in relation to aphasia characteristics. Statistical analyses included both within-group and between-group comparisons as well as correlational analyses. Results: Individuals with aphasia scored significantly higher in terms of their macrolinguistic abilities when multimodal scoring was considered compared to verbal-only scoring. Within the aphasia group, there were prominent differences noted in macrolinguistic scores for both fluent and nonfluent aphasia. Specifically, both groups scored higher on Main Concepts when multimodal scoring was considered, with the nonfluent group demonstrating significantly higher Main Concept and total macrolinguistic rubric scores in multimodal scoring compared to verbal scoring on the storytelling task. Additionally, aphasia severity showed moderate positive correlations with total macrolinguistic scores, indicating that individuals with less severe aphasia tended to produce higher quality narratives. Lastly, although persons with aphasia used different types of nonverbal modalities (i.e., drawing, writing), the use of meaning-laden gestures was most predominant during storytelling, emphasizing the importance of multimodal elements in communication for individuals with aphasia. Conclusion: Our preliminary study findings underscore the importance of considering multimodal communication in assessing discourse performance among individuals with aphasia. Tailoring assessment approaches based on aphasia subtypes can provide valuable insights into linguistic abilities and inform targeted intervention strategies for improving communication outcomes.

It's story time!: exploring the potential of multimodality in oral storytelling to support children's vocabulary learning

Exploitation Of Novel Multiplayer Gesture-Based Interaction And Virtual Puppetry For Digital Storytelling To Develop Children'S Narrative Skills

Telling stories multimodally: what observations of parent-child shared book-reading activities can bring to L2 kindergarten teachers’ training

Connection is All You Need: A Multimodal Human-AI Co-Creation Storytelling System to Support Children's Multi-Level Narrative Skills

Experience of Beauty: Valuing Emotional Engagement and Collaboration in Teacher-Child Storytelling Activities

The effects of incorporating extended conversations into video-based story retelling instruction on oral narrative skills in adolescents with intellectual disability in China

The Influence of Presentation Format of Story on Narrative Production in Chinese Children Learning English-as-a-Second-Language: A Comparison Between Graphic Novel, Illustration Book and Text

Narrativity and creativity in oral storytelling: Co-constructing a story with the audience

Exploring Cultural Identity and Language Development Through Digital Storytelling: Tales of Culturally-Linguistically Diverse Secondary Students

Storytelling as a Pedagogical Strategy to Teach English Language Vocabulary to Children

Creative and critical approaches to language learning and digital technology: findings from a multilingual digital storytelling project

The experience of story reading: deaf children and hearing mothers' interactions at story time

Narrative Intervention for Preschoolers Who Are Deaf and Hard of Hearing Using Listening and Spoken Language: A Pilot Study

Becoming fictional storytellers: African American children's oral narrative development in early elementary school

Learning Stories Through Gesture: Gesture’s Effects on Child and Adult Narrative Comprehension

“Sharing time”: Children's narrative styles and differential access to literacy

The use of storytelling techniques in extratextual activities for the development of children's expressive language skills

Visual and verbal intersections in picture books – multimodal assessment for middle years students

The relationship between young children’s language abilities, creativity, play, and storytelling

Fostering Child Language with Short-Term Digital Storybook Interventions

Expanding the scope: multimodal dimensions in aphasia discourse analysis—preliminary findings