Overview
ISO/TR 14468:2010 - "Selected illustrations of attribute agreement analysis" - is an informative Technical Report from ISO that describes how to evaluate measurement systems when the characteristics being measured are attribute data (nominal, ordinal or binary). The report provides a generic framework for Attribute Agreement Analysis (AAA), explains relevant statistics (percentage agreement, kappa, Kendall coefficients), and illustrates the methodology with five practical annex case studies (e.g., liquid crystal display manufacture, technical support triage, tasting tests, thermistor defects, assessment of disability after stroke). Note: as a Technical Report, ISO/TR 14468 is informative and intended to illustrate methods rather than impose normative requirements.
Key topics
- Scope and purpose: assessment of measurement processes where data are categorical - to judge agreement within appraisers, between appraisers, each appraiser vs a standard, and all appraisers vs a standard.
- Five-step AAA procedure (generic):
- State objectives
- Describe the measurement process (appraisers, procedures, conditions, data type)
- Design the sampling plan
- Analyse results
- Provide conclusions and suggestions
- Agreement metrics:
- Percentage of agreement (P %) - point estimate for agreement proportion (P% = n_match / n_samples × 100%).
- Kappa statistic (κ) - measures agreement beyond chance; two main variants: Cohen’s kappa (fixed appraisers) and Fleiss’s kappa (appraisers sampled from a group). Kappa ranges −1 to +1; report gives practical thresholds (κ > 0.9 excellent; κ < 0.7 needs improvement; κ < 0.4 inadequate; commonly desired κ ≥ 0.7).
- Kendall’s coefficients for ordinal data (Kendall’s W and Kendall’s tau) when magnitude/order matters.
- Hypothesis testing: use Z-statistic for κ (Z = κ / SEκ) and p-values (commonly 0.05) to test κ > 0.
Applications
ISO/TR 14468 is useful for organizations and professionals who need to validate or improve categorical measurement processes:
- Quality engineers and manufacturing (product accept/reject decisions, visual inspections)
- Six Sigma/continuous improvement teams assessing measurement system capability for categorical defects
- Clinical and sensory researchers (diagnosis agreement, taste panels, disability assessment)
- Laboratory or technical support centres for triage classification reliability
- Auditors and compliance teams verifying measurement consistency vs a standard
Practical value
- Provides a clear, systematic AAA workflow and statistical tools to quantify and interpret agreement.
- Includes real-world illustrative annexes to guide sampling, analysis and reporting.
- Helps determine whether a categorical measurement system is adequate for decision-making or needs corrective action.
Related standards
- ISO 3534-1, ISO 3534-2 (statistical vocabulary and symbols)
- References measurement system analysis (MSA) and quality management contexts (e.g., Six Sigma, ISO/TS 16949 noted in the report).