ML Workflow, Models, Neural Networks, and Development Testing

Guided briefing

ML Workflow, Models, Neural Networks, and Development Testing video briefing

A focused explanation of chapter 4, turning the AI testing theory into concrete validation checks.

Briefing focus

Module opening

This is a structured lesson briefing. Real video/audio can be added later as a media source.

Estimated time

9 min

1. 1Module opening
2. 2Learning objectives
3. 3Mind map
4. 4Scenario evidence breakdown

Transcript brief

Give QA professionals enough ML workflow knowledge to test development practices, training pipelines, and model artifacts with confidence. The briefing explains why the topic matters, walks through a failure scenario, and identifies the artefacts a tester should produce for evidence and auditability.

Key takeaways

• Connect the AI risk to a measurable test or monitor.
• Document the evidence needed for reproducibility and audit.
• Use the lab or scenario to practise the validation workflow.

Module opening

Give QA professionals enough ML workflow knowledge to test development practices, training pipelines, and model artifacts with confidence.

Audience. Testers who need to understand model creation without becoming data scientists.

Why this matters. AI testing is stronger when QA can see the training pipeline as a product: inputs, transformations, configuration, outputs, environments, and promotion controls.

ISTQB CT-AI mapping. CT-AI 1.4-1.8, 4.2, 10.1

Learning objectives

• Explain the core quality risk in ml workflow, models, neural networks, and development testing.
• Select practical test evidence that supports an AI release decision.
• Apply the module concepts to a realistic QA scenario.
• Produce a portfolio artifact that can be reused in a professional AI testing context.

Mind map

Scenario evidence breakdown

Visual flow

Topic-by-topic teaching guide

1. ML Workflow Basics

A typical workflow moves from problem framing to data preparation, training, validation, packaging, deployment, and monitoring.

2. Model Families

Different model types have different behaviours and test needs. Trees, linear models, neural networks, and LLMs fail in different ways.

3. Training Pipeline Testing

Training code should be reproducible, observable, and versioned.

4. Development Environments

AI systems often depend on packages, hardware, seeds, and data versions. Environment drift creates false confidence.

5. Model Promotion

Promotion is the controlled movement of a model into staging or production. It should require evidence, not enthusiasm.

Practical QA workflow

• Start from the user or business decision affected by the AI system.
• Name the AI asset under test: data, feature pipeline, model, prompt, retrieval index, tool, or full workflow.
• Convert the main risk into observable quality signals and release gates.
• Choose the right oracle: deterministic assertion, metric threshold, metamorphic relation, reviewer rubric, comparison, or production monitor.
• Test important slices, edge cases, misuse cases, and change scenarios.
• Record versions, data sources, thresholds, reviewer notes, and decision rationale.

Test design checklist

• What harm could happen if this AI behaviour is wrong?
• Which users, groups, products, regions, or workflows need separate evidence?
• Which metric or observation would reveal the failure early?
• What is the minimum evidence needed for release, shadow mode, rollback, or rejection?
• Who owns the evidence after the model, prompt, or data changes?

Worked QA example

A tester receives a release request for the module scenario. Instead of asking only whether tests pass, the tester writes three release questions: what changed, who could be harmed, and what evidence proves the change is controlled. The answer becomes a small evidence pack: one risk table, one set of representative examples, one automated or reviewable check, and one release recommendation.

Common mistakes

• Treating AI output as a normal deterministic response when the real risk is behavioural.
• Reporting one impressive metric without slices, uncertainty, or business context.
• Forgetting that data, prompts, model versions, and monitoring are part of the test surface.
• Writing governance language that cannot be checked by a tester.

Guided exercise

Use the scenario above and create a one-page evidence plan. Include the decision being influenced, the main risk, the test oracle, the data or examples required, the release gate, and the owner.

Discussion prompt

What should count as a release artifact: notebook, script, model binary, dataset, evaluation report, or all of them?

Hands-on lab mapping

• Lab: CourseMaterials/AI-Testing/labs/07_model_validation_metrics.ipynb
• Task: Inspect a model evaluation workflow and identify reproducibility and promotion checks.
• Why this lab matters: it turns the module theory into visible evidence that a release approver can inspect.

Decision simulation

A model performs well locally but cannot be reproduced in CI. Decide whether it can enter staging and what evidence is missing.

Key terms

• Training pipeline: Automated or semi-automated process that creates a model artifact.
• Model registry: Controlled store for versioned model artifacts and metadata.
• Reproducibility: Ability to rerun evaluation with traceable inputs and comparable results.
• Serving smoke test: A basic test that proves a packaged model loads and responds in its target environment.

Revision prompts

• Explain the module scenario in two minutes to a product owner.
• Name three pieces of evidence you would require before release.
• Identify one automated check and one human-review check.
• Describe how this topic changes after deployment.