Description

This course provides participants with the knowledge and skills necessary to become an effective member of an AI testing team. It explains the fundamental concepts of testing on AI projects including methods and practices around machine learning. We suggest that attendees hold the ISTQB Foundation Level certificate, especially if they intend to take the ISTQB AI Testing exam, but non-certificate holders can also benefit from this course.

By the end of this course, an attendee should be able to:

• Understand the current state of AI, including generative AI
• Experience the implementation and testing of machine learning models
• Understand the working and testing of simple neural networks
• Understand the specific AI quality characteristics defined by ISO/IEC 25059
• Calculate and interpret ML functional performance metrics for machine learning models
• Recognize the scope and importance of the two test levels that are specific to the testing of machine learning systems
• Contribute to the development of an effective test strategy for a machine learning system
• Design and execute test cases for machine learning systems

Outline

Introduction to Artificial Intelligence

• Introduction to AI
  • AI-Based and Conventional Systems
  • Narrow AI, General AI, and Super AI
  • Different Types of AI Technologies
  • Generative AI
  • Hardware for Machine Learning Systems
  • Development and Hosting of AI Models
  • Machine Learning Development Frameworks
  • Regulations and Standards for AI

Quality Characteristics for AI-Based Systems

• • Quality Characteristics for AI-Based Systems
    • AI-Specific Quality Characteristics
    • AI and Safety

• Acceptance Criteria for AI-Based Systems
  • Acceptance Criteria for AI-Based Systems

Machine Learning

• • Introduction to Machine Learning
    • Different Forms of Machine Learning
    • Machine Learning Workflow
    • Hands-on Exercise: Create a Machine Learning Model
    • Pretrained Models, Fine-Tuning, and Retrieval-Augmented Generation
  • Data for Machine Learning
    • Activities in Data Preparation
    • Hands-on Exercise: Data Preparation in Support of the Creation of a Machine Learning Model

• • • Hands-on Exercise: Show the Impact of Different Machine Learning Models and Dataset CombinationsML Functional Performance Metrics for Classification
      • • Calculation of Machine Learning Functional Performance Metrics
        • Hands-on Exercise: Evaluate a Machine Learning Model using Selected ML Functional Performance Metrics

• Neural Networks
  • Structure and Working of a Deep Neural Network
  • Hands-on Exercise: Experience the Implementation of a Perceptron
  • Coverage Measures for Neural Networks

Testing AI-Based Systems

• • Introduction to Testing AI-Based Systems
    • Locked and Adaptive AI-Based Systems
    • Rationale for a Statistical Approach to Testing AI-Based Systems
    • Test Oracles for AI-Based Systems

• • Testing Generative AI and Large Language Models
    • Testing Generative AI
    • Red Teaming
    • Hands-on Exercise: Exploratory Testing of a Large Language Model

• Test Levels and Machine Learning Systems
  • Test Levels for Machine Learning Systems
  • Risk-Based Testing of Machine Learning Systems

Input Data Testing for Machine Learning Systems

• Input Data Testing for Machine Learning Systems
  • Input Data Risks and Mitigations
  • Testing for Bias
  • Data Pipeline Testing
  • Testing for Data Representativeness
  • Dataset Constraint Testing
  • Label Correctness Testing
  • Hands-on Exercise: Input Data Testing

Model Testing for Machine Learning Systems

• Model Testing for Machine Learning Systems
  • Machine Learning Model Risks and Mitigations
  • Machine Learning Model Documentation and Review
  • ML Functional Performance Testing of Probabilistic Machine Learning Systems
  • Adversarial Testing of Machine Learning Systems
  • Metamorphic Testing
  • Hands-on Exercise: Apply Metamorphic Testing
  • Drift Testing
  • Testing for Overfitting and Underfitting
  • A/B Testing
  • Back-to-Back Testing

Machine Learning Development Testing

• Machine Learning Development Testing
  • Machine Learning Development Risks and Mitigations
  • Machine Learning System Deployment Testing

Prerequisites

You must have obtained an ISTQB Foundation Level Certification (CTFL) to be eligible for the AI Testing certification exam.