Users of this chapter

This chapter serves the potential needs of several disparate groups of readers. Since not all parts of this chapter will be equally relevant to every reader group, we will identify reader groups, and point out which parts of this chapter, after sections 1-3, have particular relevance for each group. The reader groups, of course, overlap to a large degree with the parties interested in speech output evaluation discussed above, but a more refined classification seems to be in order. We will distinguish the following groups of readers:

• DEVELOPERS OF SPEECH OUTPUT TESTS
  Speech output assessment is an expanding field. New tests become available in rapid succession, so that test developers want to keep abreast with what is new. Test developers will want to know the advantages and disadvantages of tests proposed in the literature, and need to know what requirements will be made to the next generation of tests. This chapter discusses many alternative tests and testing methodologies, and makes recommendations as to what type of tests are more suited for a specific purpose. The chapter will also indicate which direction speech output testing  should take in order to meet the testing requirements of the next generation of speech output systems (Section 12.6).

  Readers in this group will be interested in Section 12.5 on glass box  (diagnostic) testing  as well as in Section 12.4 on black box  testing. When systems are in their early developmental stages, glass box testing  will be most relevant; when systems have sufficiently matured, black box tests  are in order. Field testing  (Section 12.4.2) will generally be deferred until the end user groups and their specific applications are known. Field testing  will often be conducted by, or at least in close cooperation with, systems procurers for end users (see below).

  The chapter will be especially useful to those readers who are not test developers at this time but aspire at becoming test developers in the near future. These readers include, of course, students at the Ph.D.-level who want to make a career in speech output testing . A second group of readers who are new to the field are developers of speech technology in Eastern Europe and in certain third world countries where computer technology is now widely available at affordable prices, generating an immediate need for the development of speech output systems and tests in the languages of the areas concerned.

• DEVELOPERS OF SPEECH OUTPUT SYSTEMS
  A lot of speech technology research and development takes place in small high-tech companies. The research staffs are often too small to warrant the appointment of a fulltime test evaluation expert, so that a lot of diagnostic do-it-yourself testing  is going on. Developers who are not evaluation experts themselves will find this chapter a useful source of information. It identifies standard tests and test suites that are readily available for a range of diagnostic purposes (see the end of this chapter and the appendices). The remarks made above with respect to newcomers to the evaluation field apply here as well. This type of reader should concentrate on the glass box approach  (Section 12.5).

• PROCURERS OF SPEECH OUTPUT SYSTEMS
  At the most user-oriented end of the spectrum, procurers of systems will find our chapter of interest. Procurers, who themselves will more often than not be inexperienced to the field of speech output technology, will not normally be interested in diagnostic testing . They will be looking for a single figure of merit  on the basis of which to decide on the system that is best for a given range of applications. This reader group is most difficult to deal with since their needs are most divergent. There are no off-the-shelf tests that satisfy their needs. Rather we will provide numerous examples which may serve as guidelines on how to go about field testing  speech output systems for specific applications. This type of reader should concentrate on those parts of this chapter dealing with black box output testing   and field tests  (Section 12.4).

Given this division of reader groups we will present two types of recommendation, if and when we can.

• The first type suggests what decisions can be made in the present situation with what is available today, or can be made available with little effort in the immediate future. These recommendations will be found throughout this chapter, in concise format and numbered.
• The second type of recommendation that we will be making, outlines possible courses of test development for the mid and long term. Such recommendations, predominantly aimed at the evaluation experts, will be presented in Section 12.6 at the end of this chapter, in less explicit format; they will not be numbered.