Incremental effects of scene-sentence mismatch

The Carpenter and Just model (1975) predicts that for picture-sentence integration, corresponding parts of sentence and picture are serially compared with one another. Underwood et al. (2004) fail to find a mismatch effect in total sentence reading times for serial picture-sentence presentation, a result that contradicts the Carpenter and Just predictions. In two experiments (eye-tracking reading in serial scene-sentence presentation, and eye-tracking in scenes during utterance comprehension) fine-grained word region analyses revealed an incremental integration of scene and sentence both during reading and spoken comprehension, emphasizing the importance of eye-movements in investigating the incremental integration of scene and sentence. We suggest the absence of a mismatch effect in Underwood et al. is due to the grain size of their analyses (total sentence reading times only).

For relevant publications see:

Knoeferle, P., Urbach, T., & Kutas, M. (in press). Comprehending how visual context influences incremental sentence

comprehension: insights from ERPs and picture-sentence verification. Psychophysiology.

Knoeferle, P., & Crocker, M. W. (2005). Incremental Effects of Mismatch during Picture-Sentence Integration: Evidence from Eye-tracking (Talk). In: Proceedings of the 26th Annual Conference of the Cognitive Science Society. pdf of final draft