Abstract:Research in natural language processing commonly assumes that approaches that work well for English and and other widely-used languages are "language agnostic". In high-resource languages, especially those that are analytic, a common approach is to treat morphologically-distinct variants of a common root as completely independent word types. This assumes, that there are limited morphological inflections per root, and that the majority will appear in a large enough corpus, so that the model can adequately learn statistics about each form. Approaches like stemming, lemmatization, or subword segmentation are often used when either of those assumptions do not hold, particularly in the case of synthetic languages like Spanish or Russian that have more inflection than English. In the literature, languages like Finnish or Turkish are held up as extreme examples of complexity that challenge common modelling assumptions. Yet, when considering all of the world's languages, Finnish and Turkish are closer to the average case. When we consider polysynthetic languages (those at the extreme of morphological complexity), approaches like stemming, lemmatization, or subword modelling may not suffice. These languages have very high numbers of hapax legomena, showing the need for appropriate morphological handling of words, without which it is not possible for a model to capture enough word statistics. We examine the current state-of-the-art in language modelling, machine translation, and text prediction for four polysynthetic languages: Guaran\'i, St. Lawrence Island Yupik, Central Alaskan Yupik, and Inuktitut. We then propose a novel framework for language modelling that combines knowledge representations from finite-state morphological analyzers with Tensor Product Representations in order to enable neural language models capable of handling the full range of typologically variant languages.

Predicting semi-regular patterns in morphologically complex words

Complexity Trade-off in Morphosyntactic Module: Suggestions from Japanese Dialects

Joint Semantic Synthesis and Morphological Analysis of the Derived Word

Linguistically inspired morphological inflection with a sequence to sequence model

Learning to Predict Novel Noun-Noun Compounds

Morph-fitting: Fine-Tuning Word Vector Spaces with Simple Language-Specific Rules

Predicting Syntactic Choice in Mandarin Chinese: a Corpus-Based Analysis of Ba Sentences and SVO Sentences

The Structural Complexity of Chinese Words and Its Relationship with Word Frequency.

Evaluating Morphological Compositional Generalization in Large Language Models

Language learning and innateness: some implications of Compounds Research

Principles of semantic and functional efficiency in grammatical patterning

Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction

Indicatements that character language models learn English morpho-syntactic units and regularities

The Greedy and Recursive Search for Morphological Productivity

What do character-level models learn about morphology? The case of dependency parsing

The Processing Mechanism of Morphologically Complex Words for Second Language Learners

Unsupervised Learning of Morphology without Morphemes

LLMs' morphological analyses of complex FST-generated Finnish words

Morpho-syntactic lexical generalization for CCG semantic parsing

Neural Polysynthetic Language Modelling

A (distributional) semantic perspective on the processing of morphologically complex words