next up previous contents index
Next: Test material selection Up: Experimental design of large Previous: Development test


Dry run

In the ARPA  style of competitive evaluation, a special test before the actual evaluation is defined, which is identical to the actual evaluation test, but results are not made public. This test is in fact a rehearsal for the evaluation test and includes all actions to be performed by the organiser of the test and the participating laboratories. Again, the test material  should be different from the assessment test (different speakers, utterances, etc.) but all recording conditions, file formats, file naming, media types, submission protocols, etc. should be as much the same as possible.

In the EC project SQALE , where assessment of 4 languages with 4 different recognition systems is performed, the functionalities of the dry run and that of the development test  are combined into one single test. This implies that the period between the development test  or dry run and the evaluation test must be long enough to allow for system development.  

EAGLES SWLG SoftEdition, May 1997. Get the book...