The coordinated interplay account (CIA)

Dimensions of scene-sentence interaction A recent line of my research has investigated the nature of the interaction between scene, utterance, and world knowledge. Prior research on adult comprehension has shown a tight time lock between spoken comprehension and attention to relevant objects. We further know that diverse informational sources (linguistic and world knowledge as well as depicted events) rapidly influence online utterance comprehension. These insights allow us to identify two "dimensions" of comprehension in scenes (Knoeferle & Crocker, 2006):

•"Temporal dimension"

•"Informational dimension"

Experimental findings for adult comprehension Two eye-tracking experiments investigated in more detail the temporal and informational dimensions for adult comprehension. A study on the English MV/RR ambiguity corroborated the view that there is a close temporal coordination between when the verb mediates which (of two) depicted events is relevant for comprehension, and the point in time when that event influences comprehension (Experiment 1).

Experiment 2 investigated the relative importance of depicted (non-stereotypical) events compared with stereotypical knowledge of (non-depicted) events. Prior studies have shown that people can rely on either linguistic and world knowledge or depicted events when the utterance uniquely identifies either of these information types as relevant. My findings add the insight that people prefer to rely on depicted events over stereotypical thematic role knowledge for incremental thematic interpretation when those information types conflict (Knoeferle & Crocker, 2006).

The coordinated interplay account (CIA) Based on prior and own findings, we outlined a first coordinated interplay account for comprehension of utterances that relate to scenes:

1.Utterance comprehension guides attention in the scene, establishing reference to objects (Tanenhaus et al., 1995), and anticipating likely referents (see Altmann & Kamide, 1999).

2.Once the utterance has identified the most likely object or event, and attention has shifted to it, the attended scene information rapidly influences comprehension (Knoeferle & Crocker, 2006).

3.The CIA further assumes that the close time lock between utterance comprehension and attention involves a strategy of first checking the scene rather than solely relying on linguistic/world knowl- edge. Such a strategy explains the greater relative priority of immediately depicted verb-mediated events over verb-based knowledge of stereotypical events (Knoeferle & Crocker, 2006).

Further studies tested the claims of the CIA and extended the account.

For relevant publication see

Knoeferle, P. & Crocker, M.W. (2006). The coordinated interplay of scene, utterance, and world knowledge: evidence from eye tracking Cognitive Science, 30, 481-529.

Knoeferle, P., & Crocker, M. W. (2004a). Stored knowledge versus depicted events: what guides auditory sentence comprehension? (Talk) In: Proceedings of the 25th Annual Conference of the Cognitive Science Society (pp. 714-719), Mahwah, NJ: Lawrence Erlbaum. pdf of final draft