How Pre-Training Data Shapes Morphosyntactic Behavior: An Empirical View By Aiko Murakami | April 15, 2025 If you want to know why a model keeps tripping over dative alternation in German or ergative alignment in Hindi, don’t start with the attention heads. Start with the pre-training corpus. The composition of that data—which languages show up, …
Continue reading How Pre-Training Data Shapes Morphosyntactic Behavior: An Empirical View
How Pre-Training Data Shapes Morphosyntactic Behavior: A Distributional View
When a model spits out a sentence with mangled grammar, it’s easy to shrug and call it a glitch. But look closer. The error usually isn’t random—it’s a fossil of the pre-training corpus, a direct reflection of what the model saw, how often it saw it, and in what contexts. This piece digs into that …
Continue reading How Pre-Training Data Shapes Morphosyntactic Behavior: A Distributional View
How Pre-Training Data Shapes Model Behavior: A Cross-Linguistic View
Introduction: The Data Diet of a Language Model When a model handles English syntax with ease but stumbles on a simple German relative clause, the problem usually isn’t the architecture. It’s the training data. The mix of languages, the types of texts, and the presence or absence of specific grammatical patterns in the pre-training corpus …
Continue reading How Pre-Training Data Shapes Model Behavior: A Cross-Linguistic View
How Pre-Training Data Shapes Language Model Behavior
How Pre-Training Data Shapes Language Model Behavior When a language model generates text, it draws on patterns absorbed during pre-training—the massive, initial phase where the model ingests a curated collection of documents. The composition of that collection, from the sources chosen to the proportions of different text types and the decisions about what to exclude, …
Continue reading How Pre-Training Data Shapes Language Model Behavior
Why Discourse Coherence Remains the Hardest Problem in NLP — and What External Scaffolding Can and Cannot Do About It
Why Discourse Coherence Remains the Hardest Problem in NLP — and What External Scaffolding Can and Cannot Do About It Consider two paragraphs: Maria set the ancient vase on the wobbly table. It had been in her family for six generations. The wood groaned under the weight. The detective examined the scene. Maria had told …
Continue reading Why Discourse Coherence Remains the Hardest Problem in NLP — and What External Scaffolding Can and Cannot Do About It