Every language model carries a linguistic fingerprint from its pre-training corpus. Not a design choice—a statistical residue. Patterns of word order, morphological complexity, and agreement seep in without anyone explicitly programming them. For those of us who study morphosyntax across languages, the real question isn’t whether pre-training data matters. It’s how tightly it constrains the …
Continue reading When Training Data Shapes Syntax: A Cross-Linguistic Look at Model Behavior
Author:Terri Lane
How Pre-Training Data Shapes Morphosyntactic Behavior: An Empirical View
How Pre-Training Data Shapes Morphosyntactic Behavior: An Empirical View By Aiko Murakami | April 15, 2025 If you want to know why a model keeps tripping over dative alternation in German or ergative alignment in Hindi, don’t start with the attention heads. Start with the pre-training corpus. The composition of that data—which languages show up, …
Continue reading How Pre-Training Data Shapes Morphosyntactic Behavior: An Empirical View
How Pre-Training Data Shapes Model Behavior: A Cross-Linguistic View
Introduction: The Data Diet of a Language Model When a model handles English syntax with ease but stumbles on a simple German relative clause, the problem usually isn’t the architecture. It’s the training data. The mix of languages, the types of texts, and the presence or absence of specific grammatical patterns in the pre-training corpus …
Continue reading How Pre-Training Data Shapes Model Behavior: A Cross-Linguistic View
How Pre-Training Data Shapes Language Model Behavior
How Pre-Training Data Shapes Language Model Behavior When a language model generates text, it draws on patterns absorbed during pre-training—the massive, initial phase where the model ingests a curated collection of documents. The composition of that collection, from the sources chosen to the proportions of different text types and the decisions about what to exclude, …
Continue reading How Pre-Training Data Shapes Language Model Behavior
Why Discourse Coherence Remains the Hardest Problem in NLP — and What External Scaffolding Can and Cannot Do About It
Why Discourse Coherence Remains the Hardest Problem in NLP — and What External Scaffolding Can and Cannot Do About It Consider two paragraphs: Maria set the ancient vase on the wobbly table. It had been in her family for six generations. The wood groaned under the weight. The detective examined the scene. Maria had told …
Continue reading Why Discourse Coherence Remains the Hardest Problem in NLP — and What External Scaffolding Can and Cannot Do About It