Test Oracles, Metamorphic Testing, Back-to-Back Testing, and A/B Testing

Guided briefing

Test Oracles, Metamorphic Testing, Back-to-Back Testing, and A/B Testing video briefing

A focused explanation of chapter 6, turning the AI testing theory into concrete validation checks.

Briefing focus

Module opening

This is a structured lesson briefing. Real video/audio can be added later as a media source.

Estimated time

9 min

1. 1Module opening
2. 2Learning objectives
3. 3Mind map
4. 4Scenario evidence breakdown

Transcript brief

Teach AI-specific test design techniques for systems where exact expected outputs are unavailable or unstable. The briefing explains why the topic matters, walks through a failure scenario, and identifies the artefacts a tester should produce for evidence and auditability.

Key takeaways

• Connect the AI risk to a measurable test or monitor.
• Document the evidence needed for reproducibility and audit.
• Use the lab or scenario to practise the validation workflow.

Module opening

Teach AI-specific test design techniques for systems where exact expected outputs are unavailable or unstable.

Audience. Manual and automation testers designing reliable tests for probabilistic systems.

Why this matters. AI often lacks one perfect expected answer. Good testers design alternative oracles: relations, comparisons, thresholds, reviewers, and production experiments.

ISTQB CT-AI mapping. CT-AI 8.7, 9.4, 9.5

Learning objectives

• Explain the core quality risk in test oracles, metamorphic testing, back-to-back testing, and a/b testing.
• Select practical test evidence that supports an AI release decision.
• Apply the module concepts to a realistic QA scenario.
• Produce a portfolio artifact that can be reused in a professional AI testing context.

Mind map

Scenario evidence breakdown

Visual flow

Topic-by-topic teaching guide

1. The Oracle Problem

A test oracle tells us whether behaviour is acceptable. AI systems often need richer oracles than exact expected values.

2. Metamorphic Testing

Metamorphic testing checks relationships that should hold when inputs change in controlled ways.

3. Back-to-Back Testing

Back-to-back testing compares two versions, models, prompts, or providers on the same cases.

4. A/B Testing

A/B tests measure live impact with controlled exposure and guardrail metrics.

5. Human Review Rubrics

For subjective outputs, reviewers need clear criteria and examples.

Practical QA workflow

• Start from the user or business decision affected by the AI system.
• Name the AI asset under test: data, feature pipeline, model, prompt, retrieval index, tool, or full workflow.
• Convert the main risk into observable quality signals and release gates.
• Choose the right oracle: deterministic assertion, metric threshold, metamorphic relation, reviewer rubric, comparison, or production monitor.
• Test important slices, edge cases, misuse cases, and change scenarios.
• Record versions, data sources, thresholds, reviewer notes, and decision rationale.

Test design checklist

• What harm could happen if this AI behaviour is wrong?
• Which users, groups, products, regions, or workflows need separate evidence?
• Which metric or observation would reveal the failure early?
• What is the minimum evidence needed for release, shadow mode, rollback, or rejection?
• Who owns the evidence after the model, prompt, or data changes?

Worked QA example

A tester receives a release request for the module scenario. Instead of asking only whether tests pass, the tester writes three release questions: what changed, who could be harmed, and what evidence proves the change is controlled. The answer becomes a small evidence pack: one risk table, one set of representative examples, one automated or reviewable check, and one release recommendation.

Common mistakes

• Treating AI output as a normal deterministic response when the real risk is behavioural.
• Reporting one impressive metric without slices, uncertainty, or business context.
• Forgetting that data, prompts, model versions, and monitoring are part of the test surface.
• Writing governance language that cannot be checked by a tester.

Guided exercise

Use the scenario above and create a one-page evidence plan. Include the decision being influenced, the main risk, the test oracle, the data or examples required, the release gate, and the owner.

Discussion prompt

Where would exact assertions help, and where would they make your AI tests fragile?

Hands-on lab mapping

• Lab: CourseMaterials/AI-Testing/labs/06_using_ai_for_testing_playwright.ipynb
• Task: Design metamorphic checks and back-to-back comparison cases for an AI-assisted workflow.
• Why this lab matters: it turns the module theory into visible evidence that a release approver can inspect.

Decision simulation

A new model changes 30% of outputs but improves one headline metric. Decide what comparison evidence you need before approval.

Key terms

• Test oracle: A mechanism for deciding whether behaviour is acceptable.
• Metamorphic relation: A property expected to hold across related inputs.
• Back-to-back testing: Comparison of two systems or versions on the same cases.
• Guardrail metric: A metric used to prevent unacceptable side effects during an experiment.

Revision prompts

• Explain the module scenario in two minutes to a product owner.
• Name three pieces of evidence you would require before release.
• Identify one automated check and one human-review check.
• Describe how this topic changes after deployment.