Every language system starts with a corpus—a pile of texts that is, for all practical purposes, its only window into human communication. What goes into that pile matters enormously. Parliamentary transcripts, recipe blogs, old novels, forum rants: the mix determines which syntactic patterns stick, which semantic associations form, and which pragmatic conventions the system ends …
Continue reading How Pre-Training Data Shapes Language Understanding: A Computational Linguist’s View
Category:Main
How Pre-Training Data Shapes Model Behavior: A Measured Look at Corpus Effects
Every language model starts not with a clever bit of code, but with a library. The collection of texts—the pre-training corpus—is the model’s only window into human language. Before any fine-tuning, before any reinforcement learning from human feedback, the model is a statistical mirror of that corpus. It learns which words tend to follow others, …
Continue reading How Pre-Training Data Shapes Model Behavior: A Measured Look at Corpus Effects
What a Model Learns Before It Learns Anything Else: The Hidden Influence of Training Data
When a language system generates text, it isn’t reaching for some pristine, universal grammar. It’s echoing the statistical grooves worn into its training data. The makeup of that initial corpus—what’s kept, what’s discarded, and how heavily each piece is weighted—shapes everything: the vocabulary it reaches for, the facts it can retrieve, the styles it can …
Continue reading What a Model Learns Before It Learns Anything Else: The Hidden Influence of Training Data
How Models Generate Coherent Narrative Without Event Representation
The Illusion of Narrative Competence Consider a deceptively simple experiment. You give a large language model a short, carefully constructed narrative: “Elias woke up at dawn. He brewed a pot of coffee and drank a cup while reading the news. At 7:30 AM, he received a worrying email from his boss. Distracted by the email, …
Continue reading How Models Generate Coherent Narrative Without Event Representation
Why Screenplay Generation Exposes the Absence of Propositional Commitment in Language Models
Picture a scene. A character named Vera mentions a locked door in the basement. She tells her companion not to go near it. Cut to a diner. Cut back to the house — but the narrative has moved on to a phone call. By scene twelve, someone opens the door. In a competently written screenplay, …
Continue reading Why Screenplay Generation Exposes the Absence of Propositional Commitment in Language Models