Overview
ISO/TR 19358:2002 - Ergonomics: Construction and application of tests for speech technology is a Technical Report that provides guidance on designing, conducting and interpreting tests for speech-related products and services. It is an informative standard for specialists in speech technology, purchasers and end users who need reliable methods for assessing performance of automatic speech recognition (ASR), text‑to‑speech (TTS), speaker recognition/verification and related dialogue or language‑processing components.
Keywords: ISO/TR 19358:2002, speech technology testing, speech recognition evaluation, speech synthesis testing, speaker verification.
Key Topics
- Scope and terminology: Defines terms such as ASR, speech recognizer, speech synthesis, speaker identification/verification, objective and subjective assessment.
- Description of speech technologies: Covers ASR, speaker identification/verification, speech synthesis (canned vs. text‑to‑speech), speech understanding and dialogue management.
- Relevant variables for testing: Guidance on specifying speech type, speaker characteristics, task/domain, training/enrolment, environment/acoustic conditions, input/transmission paths and system modules.
- Assessment methods:
- Laboratory vs. field evaluation
- Objective (using prerecorded corpora) vs. subjective (human subjects) methods
- System transparency and diagnostic testing
- Use of corpora and related data sources for repeatable tests
- Performance measures and examples: Informative annexes include examples of assessments and a discussion of performance measures to aid interpretation of results.
Practical Applications
This Technical Report is practical for:
- Developers and researchers designing and validating ASR, TTS, speaker recognition and spoken‑dialogue systems.
- Test engineers and QA teams creating reproducible test protocols, corpora and acceptance criteria.
- Purchasers and system integrators evaluating vendor claims, comparing systems under realistic conditions (lab vs. field) and defining procurement test requirements.
- Usability and ergonomics specialists assessing user interaction, dialogue management and human factors in speech interfaces.
Use cases include command-and-control interfaces, telephone services and IVR systems, dictation and document generation, spoken‑document retrieval, and security applications using speaker verification.
Related Standards
- ISO 9921 - human-to-human speech communication (complements ISO/TR 19358 by covering interpersonal speech ergonomics).
- EAGLES Handbooks - detailed evaluation resources and standards for spoken language systems referenced in the Technical Report.
ISO/TR 19358:2002 is an authoritative reference for constructing standardized, ergonomic test procedures and interpreting performance results across speech‑technology applications.