next up previous contents index
Next: Assessment framework Up: Evaluation Previous: Background


Characterisation is a very important part of the system evaluation process. It is vital to define precisely what it is that is being evaluated, together with all the conditions under which the evaluation takes place. We therefore devote attention here to characterising the dialogue system, the task, the user, the environment , the corpus, and the overall system.

Dialogue system characterisation

Dialogue systems may be characterised by several parameters, defining their complexity; these are listed in appropriate categories below.

Knowledge databases

Language model:
  the model used by the recogniser  and shared by the system in order to guide the recognition process, whenever necessary.gif

Recogniser complexity:
isolated word, word spotting,  continuous speech  (error-free read text  or spontaneous  with repairs, hesitations, ill-formed or incomplete sentences).

the list of allowed words.

Phonological rules:
rules of pronunciation of the words.

Syntactic rules:
descriptions of the well-formed linguistic constructions recognised by the system.

Semantic-pragmatic representation :
the list of concepts used with associated structures (frames, conceptual graphs, etc.).

Task model:
  plans and scenarios , goals and subgoals, representation of the objects of the task and of their characteristics. (For example, in an air-traffic control domain, known objects would include planes with parameters such as heading, level, etc., and known goals would include authorising route adjustments, etc.).

Dialogue grammar :
hierarchy of subdialogues with dialogue act leaf nodes , indicating permitted progression from one turn  type to the next.

User model:
  particular rules of pronunciation (confusion matrix ), linguistic behaviour (particular formulations), user beliefs and knowledge about the task, etc.

System model:
  the list of the media available, with a description of their characteristics.

Dialogue strategies


The following are alternative classes of dialogue strategy which may be adopted. Considerable scope exists for further subclassification here.

Strictly guided and deterministic :
no initiative left to the user. IVR   systems typically fall into this category.

  includes correction and prediction mechanisms, shares initiative with the user, accepts interruptions or negotiation, capable of clarifying the system's choices and responses, (turn -taking is balanced between the user and the system)

  (for educational systems) the system has to learn new notions in its normal operation.

  takes into account the dynamic user model by learning the users communicative strategies and adjusting to them as each dialogue proceeds.


Task characterisation

There is an intimate connection between the application domain and associated tasks, the language required to accomplish these tasks in this domain, and the design of a system which supports dialogues for this purpose.

Task type

A broad categorisation of task types can be made, depending on whether the objects of the task are evolving during the dialogue or not. These include the following:

Information access and retrieval:
for example, train or flight time table enquiries.

the system acts as an expert-system, trying to find the best solution, for instance the best way to assign conference delegates to hotels, taking hotel costs and proximity to the conference centre into account. (An information retrieval system may need some kind of negotiation, for example, to obtain a less expensive travel ticket).

Process control:
the task is evolving, as for instance in communication with a robot.

  knowledge acquisition by the user or by the machine. In such cases as air-traffic control training , the task may be evolving (planes are changing heading or level).

  the system does not play an active part in the dialogue, but monitors its progress and is available to offer assistance when called upon. Such systems are sometimes referred to as computer mediated (or supported or assisted) human-human  communication systems. The most notable example of such a system is the VERBMOBIL   face-to-face spoken language translation system,   currently under development.

Task complexity

Tasks and the dialogues by which these tasks are achieved are more or less mutually dependent. Typically, simple tasks will be solved by means of simple dialogues and complex tasks will be accomplished by means of complex dialogues. Thus an important part of the characterisation of the dialogue system is an index of the complexity of the task or tasks to be addressed by the system. Such indices might include the following subcomponents.

User characterisation

A large number of different criteria must be taken into account when characterising users of interactive dialogue systems.  At least the following must be considered:

Environment characterisation


The environment is the total context in which an interactive dialogue system is evaluated. In general, it would be impossible to produce an exhaustive description of an environment , but a restricted set of relevant features may be selected usefully. For example, the following features relating to the acoustic environment  might be used.

Other relevant environment features will also have to be developed. As a general rule, the more features of the environment which can be characterised, the better.  

Result corpus characterisation

A system evaluation will result in the collection of a corpus of resulting dialogues. Corpora must be fully characterised to ensure that changes in system performance over time can be tracked, and that corpora collected using different systems can be reliably compared.

At least the following features of an evaluation corpus should be noted:

Overall system characterisation

The overall system in which the dialogue system is embedded also needs to be characterised. First, though, it is necessary to clarify some terms.

A mode refers to perception senses which allow for communication: the following modes may be identified: vocal, visual, auditive, tactile, olfactive.

Communication means  (or media) refer to materials or devices which are used by the dialogue system to communicate with the user.

Communication modalities  concern the way the communicating agent/party uses a mode: for speech, different modalities may be identified, for example whether continuous speech  or isolated words  are used, whether a whispering or shouting style is used, etc.

The system may comprise different communication means which may be characterised by:

Each medium has an associated language model  and is characterised by:

This characterisation is particularly important in multimodal dialogues, as the system's awareness of the states of each of the media (active, available, occupied, etc.), at each step of the dialogue, is determinant in the system predicting which media will be used by the user, or choosing the adequate media to send information to the user.

next up previous contents index
Next: Assessment framework Up: Evaluation Previous: Background

EAGLES SWLG SoftEdition, May 1997. Get the book...