Generating novel experimental hypotheses from language models: A case study on cross-dative generalization

Abstract

Neural network-based language models (LMs) have been shown to successfully capture complex linguistic knowledge. However, their utility for understanding human language and more broadly, human cognition, is still debated. We contribute to this debate by presenting a case study where we use LMs as simulated learners to derive novel experimental hypotheses that may then be tested with human learners. We apply this paradigm to study productive generalization of novel verbs across two dative constructions (DO: she pilked me the ball; PP: she pilked the ball to me) — acquisition of which is known to involve a large hypothesis space of contextual features — using LMs trained on transcripts of child-directed speech. We specifically ask: “what properties of the training exposure facilitate a novel verb’s generalization to the (unmodeled) alternate construction?” To answer this, we systematically vary the exposure context in which a novel dative verb occurs in terms of the properties of the theme and recipient, and then analyze the LMs’ usage of the novel verb in the unmodeled dative construction. As a precondition to determine their utility for deriving novel hypotheses, we first test experimental conditions analogous to prior human studies, where we find LMs to replicate known patterns of children’s cross-dative generalization. Our subsequent simulations reveal a nuanced role of the features of the novel verbs’ exposure context on the LMs’ cross-dative generalization. In particular, we find generalization to be facilitated when the first postverbal argument of the exposure context is pronominal, definite, short, and conforms to the prototypical animacy expectations of the exposure dative (i.e., inanimate theme/animate recipient). These patterns are characteristic of harmonic alignment in dative constructions observed in prior work, where the argument with features ranking higher on the discourse prominence scale tends to precede the other. This gives rise to a novel hypothesis that cross-dative generalization is facilitated or preempted based on the extent to which the features of the exposure context—in particular, its first postverbal argument—are harmonically aligned. Based on these findings, we sketch out future experiments that can test this hypothesis in child learners.