next up previous contents index
Next: Non-linguistic and other phenomena Up: The levels and types Previous: Physical transcription

Prosodic transcription


It is possible to mark some prosodic information even in orthographic transcriptions , such as lengthening of sounds, pauses in words and utterances, emphatic stress , and intonational  boundaries. Examples are the ATIS   and Switchboard corpora , and the Dutch Speech Styles Corpus. If more detail than this is required, however, it is necessary to undertake a full prosodic transcription. For an overview of some existing prosodic transcription approaches, see [Llisterri (1994)]. The following will describe prosodic transcription in general terms.

Types of approach to prosodic labelling


The above discussion has been in terms of segmental labelling only. It is also possible to annotate a speech database at the prosodic (suprasegmental) level.   This is less straightforward than segmental annotation,  as there are far fewer clear acoustic cues to prosodic phenomena. The F0   curve will be the relevant acoustic display, possibly augmented by the intensity  curve. The waveform is a useful guide to the current location in the speech and is usually displayed together with the F0 curve (as in the WAVES labelling software).

The units segmented will depend on the particular theoretical bias underlying the given research programme. A basic distinction may be drawn between a prosodic labelling system that annotates the boundaries of units (analogous to the method used in segmental annotation)  and a system that annotates the occurrence of isolated prosodic events, such as F0  peaks.

The first type of method may possibly use the intonational  categories proposed by [Nespor & Vogel (1986)], such as intonational phrase, phonological phrase, phonological word , foot , and syllable . Alternatively, it could mark the more traditional units of ``minor tone-unit'' and ``major tone-unit'', as in the MARSEC database  [Roach et al. (1993)]. Or again, it could annotate the perceptual phonetic categories used in the ``Dutch school'' of intonation  studies, such as rises and falls that are early, late or very late in their timing, fast or slow in their rate of change, and full or half sized ['t Hart et al. (1990)]. This type of annotation  could be used in conjunction with annotation  at the morphosyntactic level  to yield information about the relationship between the syntactic  and prosodic levels in terms of duration , pauses, etc.

The second type of method, though it may refer to the units mentioned above in its underlying theory, does not in fact annotate them but rather marks the occurrence of high and low tones  of various kinds. The recently formulated ToBI  transcription system [Silverman et al. (1992)] is the most well-known system of this kind for English, where the prosodic units are annotated at the ``break index'' level rather than the ``tone'' level.  (For an account of prosodic labelling for German see [Reyelt et al. (1996)]).  Other systems, such as SAMPROSA (see Appendix B) have also been proposed.

Examples of the two types of approach

Prosodic annotation  has only recently come into favour in the field of speech and language technology research. Now that a basic level of competence has been achieved as regards the synthesis  and recognition of speech segments, researchers have come to realise that much more work is required on the prosodic aspect of speech technology. This is the motivation for the growth in popularity of speech database research, and for the formulation of the ToBI   prosodic transcription system. In order for the prosodic transcriptions of various different speech databases to be comparable, and in order to make the best use of existing resources, the originators of ToBI  (Silverman et al., op. cit.) proposed a simple system that would be easy to learn and that would lead to good inter-transcriber consistency. To date it has largely been used for English, especially American English, but at least in principle it could be extended to other languages as well. The system has certain severe limitations (e.g. it has no way of representing pitch  range) but its minimalist formulation was dictated by the need for learnability and consistency in use. The ``British school'' type of system used in the MARSEC database  of British English [Roach et al. (1993)] contains more phonetic detail but may require more effort in teaching to novice transcribers. The ``IPO''  classification of F0   patterns ['t Hart et al. (1990)] has not yet been used systematically in the annotation  of large-scale publicly-available speech corpora, but has been used successfully in the development of speech synthesisers. 

Prosodic transcription also has obvious uses in basic linguistic research, especially since research into the suprasegmental aspects   of language is not nearly as advanced as research into the segmental aspects. As indicated above, a database annotated at the prosodic and morphosyntactic levels can provide information on the relationship between them with respect to duration  and pauses. If the segmental level is also annotated, then many possibilities open up for the study of segmental duration  in prosodic contexts. This is especially true in the case of languages other than English, where these aspects have received comparatively little attention to date.

The concept of levels of prosodic labelling  applies differently to the two different approaches to prosodic labelling outlined above. In the first case, the obvious categories would be those proposed by Nespor and Vogel (op. cit.), comprising levels of non-overlapping units each of which corresponds to one or more units on the level immediately below (e.g. phonological phrase, foot , syllable ). In the second case, the separate levels have no such intrinsic relationship to one another, but merely deal with different types of phenomena. For example, in the ToBI  system, there are separate levels for tones   and inter-word ``break indices''. The ToBI  system can be described briefly in terms of its separate levels, and is described below. The MARSEC  system will be outlined after that. The ``Dutch school'' system of IPO  will not be described in much detail, as it has not yet been used for annotation  of publicly-available speech corpora: however, extensive references are available in `t Hart et al.\ (op. cit.).

The ToBI labelling system


A recent experiment [Pitrelli et al. (1994)] used several prosodic transcribers working independently on the same speech data, comprising both read   and spontaneous American English speech.   The ToBI system was used, and a high level of consistency across transcribers was found, even given the fact that transcribers included both experts and newly-trained users of the system. This suggests that the system achieves its object of being easy to learn and to apply consistently, at least in the case of American English.

The ``orthographic'' level of the ToBI system contains the orthographic words   of the utterance (sometimes only partial words in the case of spontaneous speech  ). It is also possible to represent filled pauses (e.g. ``um'', ``er'') at this level.

The ``miscellaneous'' level may be used to mark the duration  of such phenomena as silence, audible breaths, laughter and dysfluencies. There is no exhaustive list of categories for this level, and different transcription projects may make their own decisions as to what to annotate.

The ``break index'' level is used to mark break indices, which are numbers representing the strength of the boundary between two orthographic words.   The number 0 represents no boundary (with phonetic evidence of cliticisation,   e.g. resyllabification of a consonant), and 4 represents a full intonation   phrase boundary (usually ``end of sentence'' in read speech ), defined by the occurrence of a final boundary tone after the last phrase tone. The number 3 represents an intermediate phrase boundary, defined by the occurrence of a phrase tone after the last pitch accent  , while the number 1 represents most phrase-medial word boundaries. The number 2 represents either a strong disjuncture with pause but no tonal discontinuity, or a disjuncture that is weaker than expected at a tonally-signalled full intonation  or intermediate phrase boundary.

The ``tone''  level is used to mark the occurrence of phonological tones at appropriate points in the F0  contour. The basic tones are ``L'' or ``H'' (for ``low'' and ``high''), but these may function as pitch accents  , phrase accents  or boundary tones, depending on their location in the prosodic unit. In the case of pitch accents   (which occur on accented syllables ), there may be one or two tones, and the H tone may or may not be ``downstepped''.

Information about the ToBI system and guidelines for transcribing are available on the Internet.


The MARSEC labelling system


The MARSEC project [Roach et al. (1993)] is based on the Spoken English Corpus [Knowles et al. (1995)], a corpus of British English that at the time was not time-aligned. The MARSEC project time-aligns the prosodic annotations,   the orthographic words , the grammatical tag of each word, and individual segments. The type of prosodic annotation  used is the ``tonetic stress mark '' type of system. Several types of accent  are recognised: low fall, high fall, low rise, high rise, low fall-rise, high fall-rise, low rise-fall, high rise-fall, low level, and high level. These may occur either on nuclear or on non-nuclear accented syllables . In addition, there is a distinction between major and minor tone-unit boundaries, and there is provision for ``markedly higher'' or ``markedly lower'' in perceived pitch . The tonetic stress mark type of system has been used for many years, and has been applied to many languages apart from English (the same is not true of the ToBI  system). However, no extensive attempts have yet been made to apply it in the field of speech technology.

The Spoken English Corpus comprises over fifty thousand words of broadcast British English in various styles, mostly monologues . Two transcribers prosodically annotated it in an auditory fashion, with no access to the F0   curve. They each transcribed half the corpus, but each also independently transcribed certain passages known as ``overlap'' passages, the purpose of which was to check on inter-transcriber consistency. Analysis of the overlap passages reveals that the consistency is fairly good, certainly in the case of major aspects such as location of accents  and direction of pitch movement   [Knowles & Alderson (1995)]. This result is especially encouraging in view of the fact that the transcription system used contains far more phonetic detail than does the ToBI  system.    

The IPO approach


The phonetically-based analysis of intonation  used at IPO ['t Hart et al. (1990)] has the advantage of having proved its usefulness for more than one language, and of having been successfully applied in the field of speech synthesis  (neither of these considerations apply to the ToBI  system). The analysis proceeds by modelling F0  curves in terms of straight lines that have been experimentally proved to be perceptually indistinguishable from the original (``close-copy stylisations''). This type of representation is then further simplified into ``standardised stylisations'' in terms of a small set of available contours for a given language. This type of representation has been experimentally proved to be distinguishable from the original on close listening, but yet not functionally any different from the original (i.e. the standardised stylisation is linguistically equivalent).

In the case of Dutch, there are ten basic pitch movements    (the model has also been applied to British English, German and Russian). These are the five falls and five rises, varying along the parameters of syllable  position, rate of pitch  change, and size of pitch  excursion. These ten pitch movements   are grouped into ``pitch configurations'' (of one or two pitch movements each). The pitch configurations are classified into prefixes , roots  and suffixes . Sequences of pitch  configurations are grouped into valid ``pitch contours'', which in turn are grouped into melodic families or ``intonation patterns''   (of which there are six in Dutch). These groupings are experimentally verified by listeners. The units of this analysis, at all levels, are based on speech corpora of spontaneous  and semi-spontaneous  utterances in Dutch. In contrast to the ToBI  and MARSEC systems , comparatively little effort has been put into checking inter-transcriber consistency, possibly because the detection and labelling  of this kind of phonetic unit is less problematic.  

Prosodic labelling in the VERBMOBIL project


In the VERBMOBIL project, a large database of German spontaneous speech   is being recorded at Munich, Bonn, Kiel and Karlsruhe. It covers a variety of different German speaking styles . Part of these data are being prosodically labelled at IPDS Kiel according to the VERBMOBIL  prosodic conventions PROLAB [Kohler et al. (1995)]. Another section of the corpus is being processed at Braunschweig University according to an adapted ToBI  system, along the following guidelines:

  1. The data should be usable by several project partners for a variety of purposes. Therefore the inventory of labels was designed at several workshops held with potential users of the data.
  2. It is necessary to use transcribers with little experience in prosodic labelling, in order to transcribe the amount of data needed for the training  of statistical speech recognisers .

The tasks include not only the labelling of speech data, but also the development of a workstation for prosodic labelling, and methods and tools for increasing labelling speed and consistency, as follows:

The label inventory splits into three tiers, as follows:

  1. Functional tier: main accent , secondary accent, emphasised/contrastive accent, sentence modality.
  2. Break index tier: (full) intonation  phrase boundary, minor boundary, irregular boundary.
  3. Tone tier:  pitch accents   and boundary tones.

The functional tier provides information about prosodic function like focus  and modality.

Question mark: Several question types are labelled. The recognition of questions and non-questions is one of the disambiguation tasks of prosody in VERBMOBIL.

Main accent:  In each intonational  phrase the most prominent word is labelled. There can be more than one PA per intonational  phrase, however.

Secondary accent: All other accents are secondary accents.

Emphasised or contrastive accent:  This is an emphasised (i.e. extraordinarily strong) accent.

This functional tier seems to be unique among labelling systems. The reasons for the introduction of an explicit functional tier are as follows:

  1. For a lot of users, this tier is the most important one besides the break index tier. The linguistic levels in speech recognition  tend to use this functional part directly, while other systems merely need such information as unaccented/accented/strong accent. 
  2. The functional tier can also be found implicitly in other labelling systems. However, in the case of spontaneous speech , it appears to be more appropriate to mark the prominence relations explicitly, rather than assuming them from accent  and boundary positions.
  3. Focus  information is very interesting in spontaneous speech . Although phrase accent  and focus are not identical, the phrase accent may give clues about the location of the focus (see the similarities between the Stuttgart tones and the Braunschweig PA).
  4. Previous experiments have shown that even inexperienced transcribers can mark phrase accents  with a high degree of inter-transcriber consistency.

The break index tier marks different types of word boundary,   as follows:

Normal word boundary. 
Minor (intermediate) phrase boundary. This is a weak intonational  marking, comparable to the ``trail'' tones used at the University of Stuttgart, and possibly to a ``3'' boundary in the ToBI  system.
Full intonational boundary.  This is a strong intonational event either with or without lengthening or change in speech tempo.
Irregular boundary. This boundary marks dysfluencies at hesitations, repairs etc.

The tone  tier uses a ToBI-like inventory consisting of H and L tones [Reyelt et al. (1996), see also,]. The pitch accents   and boundary tones are intended as a phonologically distinctive minimal system, together with additional distinctions which proved to be necessary for labelling spontaneous speech . The accents are as follows:

This is the ``normal'' peak accent. It occurs as a rising accent in prenuclear position, and a falling accent in nuclear position before a low boundary tone.

The auditory impression within the accented syllable  is ``high''.

This is a medium (or rather a ``raised'') peak. Beginning with a low tone , the tex2html_wrap_inline45157 rises to a high peak within the syllable . This accent  type is often combined with emphasis.

The auditory impression within the accented syllable  is ``high''.

This is a ``delayed'' peak, with a H* accent that reaches high tex2html_wrap_inline45157 in the syllable  after the accented one. The German term for it is ``Schleifton''.

The auditory impression within the accented syllable is ``rising, between low and high''.

This is a ``trough accent''. It can take a rising form when followed by a H-H% boundary.

The auditory impression within the accented syllable is ``low''.

!H*, L+!H*, L*+!H
These are downstepped accents.

This is an ``early peak''. It comprises a fall before the accented syllable , and is often followed by a low boundary.

The auditory impression within the accented syllable is ``low''.

Each intonational  phrase boundary is marked by two tones , a phrase tone and a boundary tone. These are both labelled, even if there is no clear bitonal pitch movement   (and especially at low boundaries).

This is a ``terminal fall'', where the tex2html_wrap_inline45157 reaches the bottom of the speaker's pitch range .

This is a ``question/continuation rise'', where the tex2html_wrap_inline45157 approaches the top of the speaker's pitch  range. There is a very steep rise after a low nuclear accent, or a high plateau after a H* accent where the high tones  cause tex2html_wrap_inline45157 to remain high.

This is a low phrase tone with a pitch  rise to the mid or high level. This rise must be distinguished from a general slight final rise that is idiomatic for some speakers, and is discarded.

This is a ``continuation fall''. In some cases it involves a fall from high to mid pitch , and in others a more level boundary at mid pitch .


Provisional recommendations for prosodic transcription

It is reasonable to assume nowadays that a prosodic transcriber will have access to at least the waveform and the F0  curve for the speech to be transcribed. In that case, the recommendation is to use either the ToBI  or the IPO  system (and the MARSEC system  if a purely auditory transcription is being carried out). If the language to be transcribed is not English, and especially if the projected application of the prosodic transcription is in the field of speech technology, then it is probably best to use the IPO  system if possible (i.e. if the basic ``grammar'' of contours has already been researched for that language). However, these can only be provisional recommendations, as little work has been carried out on prosodic labelling  in comparison with the great effort that has been expended on segmental labelling . In this situation, it may be that a different system entirely will prove more appropriate to the given language, and it is not possible to make absolute recommendations.   

next up previous contents index
Next: Non-linguistic and other phenomena Up: The levels and types Previous: Physical transcription

EAGLES SWLG SoftEdition, May 1997. Get the book...