The purpose for which the recogniser can be assessed covers a wide range. It varies from ``application oriented'', (how well is this system suited to a particular use, or which system is the best for my application) to ``focussed on development'' which is a diagnostic approach (what are difficult phonemes to distinguish, what are ideal parameter settings). For each purpose one or more methodologies exist.
In the SAM final report [SAM (1992)] a range of assessment methods are described:
The methodologies have been ordered from application oriented to diagnostic , thus suggesting a natural mapping towards the purpose of the assessment. The first method, using representative databases, is an expensive field experiment and difficult to reproduce, but gives the best measure for real performance. The last method, using artificial test signals , is less expensive and reproducible, but gives no absolute measure of performance. In Section 10.3 we will treat the various assessment methods in greater detail.