next up previous contents index
Next: Linguistic vs. acoustic Up: Towards a taxonomy of Previous: Glass box vs. black

 

Laboratory vs. field

   

Ideally, any speech output system should perform at the same level of adequacy as a human speaker (though there might be counterexamples where users want to know whether they are being addressed by a machine or a human; in such applications is seems advisable to use synthetic speech that is recogisably non-human). Such a system would be optimal for any application. However, given that systems available today are less than optimal, it is important to know which aspects of a system's performance are essential to a specific application. Speech output systems typically form an element of a larger human-machine interface  in an application with a specific, dedicated task. In practice this means that, quite probably, the vocabulary  and types of information exchanges are restricted and domain-specific, so that situational redundancy  is likely to make up for poor intelligibility. On the other hand, speech output systems will often be used in complex information processing tasks, so that the listener has only limited resources available for attending to the speech input. Also, end users may have different attitudes towards, and motivations for, working with artificial speech than subjects in laboratory experiments, especially when the latter have not been explicitly selected so as to be fully representative of the end users. It is often hazardous, therefore, to predict beforehand, on the basis of laboratory tests, how successful a speech output system will be in the practical application. Generally, as an application situation contains more specific aspects, less prediction of field performance is afforded by laboratory tests. Output systems will have to be tested in the field, i.e. in the real situation, with the real users. The use of field tests will be limited to one system in one specific application; results of a field test cannot, as a rule, be generalised to other systems and/or other applications.

Recommendations on choice of test methodology

  1. Do not rely on laboratory tests alone.
  2. As soon as there is a discrepancy between the laboratory setting and the true field situation (in terms of environment,  tasks, type of listener) field testing is necessary.

   



next up previous contents index
Next: Linguistic vs. acoustic Up: Towards a taxonomy of Previous: Glass box vs. black

EAGLES SWLG SoftEdition, May 1997. Get the book...