There has been relatively little work on what psychophysical procedures are appropriate for assessing speech recognisers. Fortunately, there is a second area of investigation which involves similar issues - measuring intelligibility of speech with the hearing impaired - where different psychophysical procedures have been investigated. The procedures, outlined below, require a judgment to be made to stimulus materials. They can be applied directly by language engineers interested in assessing speech output systems where the stimulus materials are synthesised speech.
The situation is more tricky when considering how the procedures might apply to speech recognition (cf. the discussion in Chapter 10): The basic problem is what is the nature of the stimulus material in these cases. One solution is for the recognisers to produce computed output which is then assessed with the procedures. The outputs of the recogniser could be in the form of phoneme labels which would constitute the stimuli. Alternatively, the phonemic output could be converted to speech form using a text to speech system, and the tests could then proceed in exactly the same way as with synthetic speech. However, it needs to be stated that this seems a roundabout procedure. Assume that human speech is read out from text/script. The recogniser also typically outputs text/orthography. Assessment could then consist of straightforward comparison of input/output (in terms of no. of correct words).
In magnitude estimation , subjects choose a positive number to represent the subjective magnitude of the intelligibility of the output of different recognisers.
The rank order procedure , as its name suggests, would require subjects to place the different recogniser outputs in order of magnitude of increasing intelligibility.
In paired comparison as applied to recogniser assessment, subjects judge which of two recogniser outputs has more or less intelligibility.