ISO IEC TR 29119-11-2020 PDF

Full title and description

Software and systems engineering — Software testing — Part 11: Guidelines on the testing of AI‑based systems. This technical report provides guidance for planning, designing and executing tests of systems that include at least one AI component, addressing AI‑specific characteristics such as non‑determinism, data dependence, interpretability and the test oracle problem.

Abstract

This technical report introduces AI‑based systems and explains characteristics that create new testing challenges (for example deep neural networks, big data dependence, poor or evolving specifications and non‑deterministic behaviour). It describes the resulting difficulties in specifying acceptance criteria (the test oracle problem), offers guidance for testing across the lifecycle, and recommends both black‑box approaches and white‑box techniques for neural networks, together with guidance on test environments and scenario derivation. An AI‑based system is defined here as any system that includes at least one AI component.

General information

• Status: Published (technical report; under periodic review by ISO/IEC).
• Publication date: November 2020 (first edition).
• Publisher: International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC).
• ICS / categories: 35.080 — Software.
• Edition / version: Edition 1 (ISO/IEC TR 29119‑11:2020).
• Number of pages: 52 pages (first edition, technical report).

Scope

This TR gives guidance for testing systems that contain AI components. It addresses characteristics of AI‑based systems (flexibility/adaptability, autonomy, evolution, bias, complexity, transparency/interpretability and non‑determinism) and explains implications for specifying acceptance criteria, designing test strategies, selecting test techniques and constructing realistic test environments and scenarios. It is intended to be applicable across the system lifecycle and to complement the broader ISO/IEC 29119 software testing series.

Key topics and requirements

• Overview of AI technologies, use cases and implications for testing (definitions, narrow vs general AI, development frameworks and hardware considerations).
• Identification of AI‑specific characteristics that affect testing (non‑determinism, bias, opacity, evolution and data dependence).
• Discussion of the test oracle problem and approaches to defining expected outcomes for AI components.
• Guidance on combining black‑box testing with white‑box techniques for neural networks and other ML models.
• Recommendations for test environments, scenario creation (including regulatory and safety scenarios) and realistic data usage.
• Considerations for testing over time: model updates, retraining, continuous monitoring and regression testing for evolving AI components.
• Practical advice for stakeholders: test planning, acceptance criteria, data quality checks and traceability.

Typical use and users

The TR is intended for test managers, QA engineers, ML engineers and data scientists, system architects, developers integrating AI components, safety/regulatory engineers, tool vendors and researchers who need structured guidance on testing AI‑based systems. It helps teams define test strategies, acceptance criteria and realistic test environments for non‑deterministic or data‑driven components.

Related standards

This report complements the ISO/IEC 29119 series (concepts/definitions, test processes, documentation, techniques and related parts) and should be used alongside other sector‑specific or safety standards when testing safety‑critical AI (for example automotive, medical or aerospace standards applicable to the system). The ISO/IEC 29119 family (and related TR/parts) provides the broader testing framework to which this AI‑focused guidance attaches.

Keywords

AI testing, AI‑based systems, software testing, ISO/IEC 29119, test oracle problem, neural network testing, black‑box testing, white‑box testing, test environments, model validation, data quality, non‑determinism, explainability.

FAQ

Q: What is this standard?

A: ISO/IEC TR 29119‑11:2020 is a technical report that provides guidelines for testing systems that include AI components, focusing on challenges unique to AI and machine learning.

Q: What does it cover?

A: It covers AI system characteristics, the test oracle problem, lifecycle testing approaches, recommended test techniques (both black‑box and white‑box for neural networks), designing test environments and deriving test scenarios, plus considerations for bias, explainability and continuous model evolution.

Q: Who typically uses it?

A: Test managers, QA and test engineers, ML engineers, data scientists, system architects, safety/regulatory assessors and tool vendors use it to inform AI test strategies and acceptance criteria.

Q: Is it current or superseded?

A: The document was published in November 2020 (first edition). It is a published ISO/IEC technical report and, like other ISO products, is subject to periodic review; as published it remains the current TR for Part 11. Users should check ISO/IEC review status for any updates or revisions after 2020.

Q: Is it part of a series?

A: Yes — it is Part 11 of the ISO/IEC 29119 family of standards on software testing and is intended to be used alongside other parts of the series that cover concepts, processes, documentation and techniques.

Q: What are the key keywords?

A: AI testing, test oracle, neural networks, non‑determinism, explainability, test environments, model validation, ISO/IEC 29119.