Alignment research is the study of how to make the goals of an artificial system match the goals of its designers and users. It sits at the intersection of technical specification, behavioral testing, and normative judgment. Adjacent concepts include value specification, reward modeling, interpretability, and robustness. For a blog that examines where language models fail …
Continue reading Why Alignment Research Needs Philosophical Rigor
Author:Terri Lane
How Pre-Training Data Determines Model Behavior
Pre-training data is the initial, large-scale text corpus used to shape a model’s statistical expectations before any task-specific adjustment. In morphosyntax research, this corpus is not a neutral substrate. It encodes frequency distributions, register biases, orthographic conventions, and cross-linguistic asymmetries that later surface as model behavior. Adjacent concepts include data mixture ratios, tokenization artifacts, corpus …
Continue reading How Pre-Training Data Determines Model Behavior
When Training Data Shapes Syntax: A Cross-Linguistic Look at Model Behavior
Every language model carries a linguistic fingerprint from its pre-training corpus. Not a design choice—a statistical residue. Patterns of word order, morphological complexity, and agreement seep in without anyone explicitly programming them. For those of us who study morphosyntax across languages, the real question isn’t whether pre-training data matters. It’s how tightly it constrains the …
Continue reading When Training Data Shapes Syntax: A Cross-Linguistic Look at Model Behavior
How Pre-Training Data Shapes Morphosyntactic Behavior: An Empirical View
How Pre-Training Data Shapes Morphosyntactic Behavior: An Empirical View By Aiko Murakami | April 15, 2025 If you want to know why a model keeps tripping over dative alternation in German or ergative alignment in Hindi, don’t start with the attention heads. Start with the pre-training corpus. The composition of that data—which languages show up, …
Continue reading How Pre-Training Data Shapes Morphosyntactic Behavior: An Empirical View
How Pre-Training Data Shapes Morphosyntactic Behavior: A Distributional View
When a model spits out a sentence with mangled grammar, it’s easy to shrug and call it a glitch. But look closer. The error usually isn’t random—it’s a fossil of the pre-training corpus, a direct reflection of what the model saw, how often it saw it, and in what contexts. This piece digs into that …
Continue reading How Pre-Training Data Shapes Morphosyntactic Behavior: A Distributional View