Modelling scene-sentence interaction

In collaboration with Marty Mayberry and Matthew W. Crocker, we have begun to model comprehension of sentences that relate to scenes. The model is based on an SRN architecture (Elman, 1990), and has been extended to include a representation of the scene (objects and depicted events). Core properties of connectionist networks such as their ability to incrementally integrate information, to develop expectations based on context, and to integrate diverse informational sources, make them particularly well-suited for modeling the incremental interplay of scene and utterance.

The network models both the rapid use of linguistic and world knowledge (e.g., Kamide et al., 2003) and the rapid use of depicted events for incremental thematic role assignment (Knoeferle et al., 2005). A further simulation has shown that connectionist networks can be trained to model recent eye tracking findings that suggest people prefer to rely on depicted events rather than stereotypical thematic role knowledge for thematic interpretation.

For relevant publication see

Crocker, M.W., Knoeferle, P., & Mayberry, M. (2010). Situated sentence processing: The coordinated interplay account and a neurobehavioral model. Brain and Language 112, 189-201. [crocker_etal_bl.pdf]

Mayberry, M., Crocker, M.W., & Knoeferle, P. (2009). Learning to Attend: A Connectionist Model of the Coordinated Interplay of Utterance, Visual Context, and World  Knowledge. Cognitive Science, 33, 449-496.

Mayberry, M., Crocker, M. W., & Knoeferle, P. (2005). A Connectionist Model of Anticipation in Visual Worlds. In: Robert Dale, Kam-Fai Wong, Jian Su, Oi Yee Kwong (eds) Proceedings of the Second International Joint Conference on Natural Language Processing (Lecture Notes in Computer Science). (pp. 849-861). Berlin: Springer.pdf of final draft

Mayberry, M., Crocker, M. W. , & Knoeferle, P. (2005). A Connectionist Model of Sentence Comprehension in Visual Worlds (Talk). In: Proceedings of the 26th Annual Conference of the Cognitive Science Society. pdf of final draft

Mayberry, M., Crocker, M.W., & Knoeferle, P. (2005). A Connectionist Model of Language- Scene Interaction. In: Proceedings of the ACL Workshop Psychocomputational Models of Human Language Acquisition.