Do you prefer audio books over printed ones, and if so, what is the advantage? Why is video conferencing so awkward? Why do you rely on SMS messages as a means of communication in some situations but on e-mailing in others? Did you ever notice that you raise your eyebrows when asking people a question – even on the telephone? And did you ever wonder how to make your point in a discussion most convincingly? Well – read on.
A5 in brief
In our research project, we aim at the empirically founded distinction of diverse modes of alignment and the assessment of the facilitatory role of alignment by systematically contrasting mediated communication (in which there are technical constraints to alignment) to face-to-face communication (in which there are no such constraints). In our investigations, we focus on visuolinguistic aspects; in particular, the relationships that hold between attention, eye gaze behavior, and the processing of verbal utterances in dialogue.
A5 in some more detail
The rationale behind our project is to contrast technically constrained to face-to-face communication situations. At that, eye movement measurement permits a fine-grained investigation of the transition between direct and mediated communication by stepwise reduction of the information available on the diverse sensory channels. Accordingly, our investigation of alignment in mediated communication proceeds in a stepwise fashion, with natural, unconstrained face-to-face communication serving as a reference. By systematically introducing media-related technical constraints on conversation, we can study the strategies that interlocutors employ in order to compensate for the reduced alignment possibilities. A fine-grained analysis of the time lag between relevant verbal and nonverbal events under the diverse conditions will provide us with clues to the cognitive processes at the bottom of alignment in communication. Last but not least, we are working on the development and evaluation of a formal model of the relationships that hold between interlocutors’ eye movements and the occurrence of particular linguistic phenomena in face-to-face and mediated dialogue.
Methodologically, our research relies on both experimentation and simulation. In our experimental studies, we measure interlocutors’ eye movements. The eye-movement data are matched to linguistic events in the ongoing dialogue, and will be supplemented by global measures such as reaction times or the occurrence of errors. In our simulative work, models of eye gaze behavior are being developed that encompass a range of relevant situational and verbal predictors. These models are then evaluated with respect to cognitive adequacy and implemented into artificial interlocutors.
For a scenario, we favor a comparison task, in which the interlocutors have to identify and discuss the similarities and differences between two seemingly identical sets of objects. A major advancement of this task is that it permits a systematic variation of various stimulus dimensions.
Thus, we pursue both basic research goals and applied aspects. Specifically, we aim at
- an extension of the interactive alignment approach to mediated communication, with emphasis on the investigation of the intricate visuolinguistic relationships that hold in dialogue.
- an in-depth examination of various forms of mediated communication (e.g., video conferencing) and hence, diverse modes of alignment, in comparison to face-to-face dialogue.
- the development of innovative experimental paradigms, tasks, and materials for dialogue research – for instance, the simultaneous eye movement tracking of two interlocutors.
- the modelling of human eye gaze patterns and alignment-related linguistic phenomena with respect to efficiency, functionality, usability, and cognitive adequacy.