Evaluation protocol

If one organises a competitive benchmarking  evaluation among several speech recognition laboratories, an evaluation test protocol should be kept. In this protocol, the coordinator should specify all details concerning the test. This is necessary, because for such an evaluation the recognition is not performed at the location of the organising institution, but at the sites of the laboratories themselves. Items that should certainly be defined in such a protocol are:

• Dates and times when the evaluation test begins, when the results are due, when the adjudication period ends, when the official results will be published.
• A description of how the recognition results are to be submitted to the organising institute, file formats, file structure, etc.