next up previous contents index
Next: Recommendations Up: Scoring procedures Previous: Verification

Open-set identification

   

An open-set identification system can be viewed as a function which assigns to any test utterance  z an estimated speaker index tex2html_wrap_inline48651, corresponding to the identified speaker tex2html_wrap_inline48653 in the set of registered speakers , or outputs 0 if the applicant speaker  is considered as an impostor.

In open-set identification, three types of error can be distinguished:

Here, two points of view can be adopted.

Either a misclassification  error is considered as a false acceptance  (while a correct identification is treated as a true acceptance ). In this case, open-set identification can be scored in the same way as verification, namely by evaluating a false rejection  rate tex2html_wrap_inline45995 and a false acceptance  rate tex2html_wrap_inline46691. The concept of ROC   curve can be extended to this family of systems, and in particular, an equal error rate  tex2html_wrap_inline48475 can be computed. However, the false acceptance  rate tex2html_wrap_inline46691 is now bounded by a value tex2html_wrap_inline48671 when the threshold tex2html_wrap_inline48431 tends to 0, tex2html_wrap_inline48675 being the closed-set misclassification  rate of the system, i.e. the performance that the open-set identification system would provide if it was functioning in a closed-set mode. Therefore, a parametric approach for dynamic evaluation would require a specific class of ROC  curve models (at least with two parameters). Moreover, merging classification errors with false acceptances  may not be appropriate if the two types of error are not equally harmful.

An alternative solution is to keep distinct the three types of error, and measure them by three rates tex2html_wrap_inline45995, tex2html_wrap_inline46691 and tex2html_wrap_inline47113. The ROC  curve is now a curve in a three-dimensional space, with equation tex2html_wrap_inline48683. The two extremities of this curve are the points with coordinates tex2html_wrap_inline48685 and tex2html_wrap_inline48687. The ROC   curve can be projected as tex2html_wrap_inline48489 and tex2html_wrap_inline48691. The first projection is a monotonically decreasing curve such as tex2html_wrap_inline48447 and tex2html_wrap_inline48449, whereas the second projection is also monotonically decreasing, and satisfies tex2html_wrap_inline48697 and tex2html_wrap_inline48699. A minimal description of the curve of tex2html_wrap_inline48701 could then be the equal error rate  tex2html_wrap_inline48475 of function f and the closed-set identification   score tex2html_wrap_inline48675 of function g. Parametric models of tex2html_wrap_inline48701 with two degrees of freedom could be thought of, but to our knowledge, this remains an unexplored research topic.

Among both possibilities, we believe that the second one is to be preferred, though it is slightly more complex.

 



next up previous contents index
Next: Recommendations Up: Scoring procedures Previous: Verification

EAGLES SWLG SoftEdition, May 1997. Get the book...