Abstract:Background: Non-word repetition (NWR) tests are an important way speech and language therapists (SaLTs) assess language development. NWR tests are often scored whilst participants make their responses (i.e., in real time) in clinical and research reports (documented here via a secondary analysis of a published systematic review). Aims: The main aim was to determine the extent to which real-time coding of NWR stimuli at the whole-item level (as correct/incorrect) was predicted by models that had varying levels of detail provided from phonemic transcriptions using several linear mixed method (LMM) models. Methods & procedures: Live scores and recordings of responses on the universal non-word repetition (UNWR) test were available for 146 children aged between 3 and 6 years where the sample included all children starting in five UK schools in one year or two consecutive years. Transcriptions were made of responses to two-syllable NWR stimuli for all children and these were checked for reliability within and between transcribers. Signal detection analysis showed that consonants were missed when judgments were made live. Statistical comparisons of the discrepancies between target stimuli and transcriptions of children's responses were then made and these were regressed against live score accuracy. Six LMM models (three normalized: 1a, 2a, 3a; and three non-normalized: 1b, 2b, 3b) were examined to identify which model(s) best captured the data variance. Errors on consonants for live scores were determined by comparison with the transcriptions in the following ways (the dependent variables for each pair of models): (1) consonants alone; (2) substitutions, deletions and insertions of consonants identified after automatic alignment of live and transcribed materials; and (3) as with (2) but where substitutions were coded further as place, manner and voicing errors. Outcomes & results: The normalized model that coded consonants in non-words as 'incorrect' at the level of substitutions, deletions and insertions (2b) provided the best fit to the real-time coding responses in terms of marginal R2, Akaike's information criterion (AIC) and Bayesian information criterion (BIC) statistics. Conclusions & implications: Errors that occur on consonants when non-word stimuli are scored in real time are characterized solely by the substitution, deletion and insertion measure. It is important to know that such errors arise when real-time judgments are made because NWR tasks are used to assess and diagnose several cognitive-linguistic impairments. One broader implication of the results is that future work could automate the analysis procedures to provide the required information objectively and quickly without having to transcribe data. What this paper adds: What is already known on this subject Children and patients with a wide range of cognitive and language difficulties are less accurate relative to controls when they attempt to repeat non-words. Responses to non-words are often scored as correct or incorrect at the time the test is conducted. Limited assessments of this scoring procedure have been conducted to date. What this study adds to the existing knowledge Live NWR scores made by 146 children were available and the accuracy of these judgements was assessed here against ones based on phonemic transcriptions. Signal detection analyses showed that live scoring missed consonant errors in children's responses. Further analyses, using linear mixed effect models, showed that live judgments led to consonant substitution, deletion and insertion errors. What are the practical and clinical implications of this work? Improved and practicable NWR scoring procedures are required to provide SaLTs with better indications about children's language development (typical and atypical) and for clinical assessments of older people. The procedures currently used miss substitutions, deletions and insertions. Hence, procedures are required that provide the information currently only available when materials are transcribed manually. The possibility of training automatic speech recognizers to provide this level of detail is raised.

Real‐time coded measures in natural language samples capture change over time in minimally verbal autistic children

The Utility of Natural Language Samples for Assessing Communication and Language in Infants Referred with Early Signs of Autism

Consistency and reliability of automated language measures across expressive language samples in autism

The utility of LENA as an indicator of developmental outcomes for young children with autism.

Remote Natural Language Sampling of Parents and Children With Autism Spectrum Disorder: Role of Activity and Language Level

Use of a Video Scoring Anchor for Rapid Serial Assessment of Social Communication in Toddlers

Predicting Language in Children with ASD Using Spontaneous Language Samples and Standardized Measures

Factors affecting judgment accuracy when scoring children's responses to non-word repetition stimuli in real time

Spoken Language Change in Children on the Autism Spectrum Receiving Community-Based Interventions

Child language and autism diagnosis impact hierarchical temporal structure of parent-child vocal interactions in early childhood

Using the LENA® system for children with autism in educational settings: A comparison with human coders

Language ENvironment Analysis (LENA) in Phelan-McDermid Syndrome: Validity and Suggestions for Use in Minimally Verbal Children with Autism Spectrum Disorder

Eliciting Language Samples for Analysis (ELSA): A New Protocol for Assessing Expressive Language and Communication in Autism

A new measure of child vocal reciprocity in children with autism spectrum disorder

Identifying Childhood Expressive Language Features That Best Predict Adult Language and Communication Outcome in Individuals With Autism Spectrum Disorder

Frequency and characteristics of echoes and self-repetitions in minimally verbal and verbally fluent autistic individuals

Automatic speaker diarization for natural conversation analysis in autism clinical trials

Bias in measurement of autism symptoms by spoken language level and non-verbal mental age in minimally verbal children with neurodevelopmental disorders

Comparing Automatic and Manual Measures of Parent-Infant Conversational Turns: A Word of Caution

Objective measurement of vocalizations in the assessment of autism spectrum disorder symptoms in preschool age children

Measuring Change in Social Interaction Skills of Young Children with Autism