Description

This course provides participants with the knowledge and skills necessary to become an effective member of testing team leveraging Generative AI. It explains the fundamental concepts of using Generative AI for testing on software projects including methods and practices around effective prompt engineering and managing risks associated with Generative AI. We suggest that attendees hold the ISTQB Foundation Level certificate, especially if they intend to take the ISTQB Testing with Generative AI exam, but non-certificate holders can also benefit from this course.

By the end of this course, an attendee should be able to:

• Understand the fundamental concepts, capabilities, and limitations of Generative AI
• Develop practical skills in prompting large language models for software testing
• Gain insight into the risks and mitigations of using Generative AI for software testing
• Gain insight into the applications of Generative AI solutions for software testing
• Contribute effectively to the definition and implementation of a Generative AI strategy and roadmap for software testing within an organization

This course prepares you for the ISTQB Testing with Generative AI exam.

Outline

Introduction to Generative AI for Software Testing

• • Generative AI Foundations and Key Concepts
    • • AI Spectrum: Symbolic AI, Classical Machine Learning, Deep Learning, and Generative AI
      • Basics of Generative AI and LLMs

    Hands-on Exercise: Practice tokenization and token count evaluation when using an LLM for a software test task

    • • Foundation, Instruction-Tuned and Reasoning LLMs
      • Multimodal LLMs and Vision-Language Models

    Hands-on Exercise: Write and execute a prompt for a multimodal LLM using both textual and image inputs for a software test task

• • Leveraging Generative AI in Software Testing: Core Principles
    • Key LLM Capabilities for Test Tasks
    • AI Chatbots and LLM-Powered Testing Applications for Software Testing

Prompt Engineering for Effective Software Testing

• • Effective Prompt Development
    • • Structure of Prompts for Generative AI in Software Testing

    Hands-on Exercise: Observe several given prompts for software test tasks, identifying the components of role, context, instruction, input data, constraints and output format in each

    • • Core Prompting Techniques for Software Testing

    Hands-on Exercise: Observe demonstrations of prompt chaining, few-shot prompting, and meta prompting applied to software test tasks
    Hands-on Exercise: Identify which prompt engineering techniques are being used in given examples

    • System Prompt and User Prompt

• • Applying Prompt Engineering Techniques to Software Test Tasks
    • • Test Analysis with Generative AI

    Hands-on Exercise: Practice multimodal prompting to generate acceptance criteria for a user story based on a GUI wireframe
    Hands-on Exercise: Practice prompt chaining and human verification to progressively analyze a given user story and refine acceptance criteria

    • • Test Design and Test Implementation with Generative AI

    Hands-on Exercise: Practice functional test case generation from user stories with generative AI using prompt chaining, structured prompts and meta-prompting
    Hands-on Exercise: Use few-shot prompting technique to generate Gherkin style test conditions and test cases from user stories
    Hands-on Exercise: Use prompt chaining to prioritize test cases within a given test suite, taking into account their specific priorities and dependencies

    • • Automated Regression Testing with Generative AI

    Hands-on Exercise: Practice few-shot prompting to create and manage keyword-driven test scripts
    Hands-on Exercise: Practice structured prompt engineering for test report analysis

    • • Test Monitoring and Test Control with Generative AI

    Hands-on Exercise: Observe test monitoring metrics prepared by generative AI from test data

    • • Choosing Prompting Techniques for Software Testing

    Hands-on Exercise: Select and apply context-appropriate prompting techniques for a given test task

  • Evaluate Generative AI Results and Refine Prompts for Software Test tasks
    • • Metrics for Evaluating the Results of Generative AI on Test tasks

    Hands-on Exercise: Observe how metrics can be used for evaluating the result of generative AI on a test task

    • • Techniques for Evaluating and Iteratively Refining Prompts

    Hands-on Exercise: Evaluate and optimize a prompt for a given test task

Managing Risks of Generative AI in Software Testing

• • Hallucinations, Reasoning Errors and Biases
    • • Hallucinations, Reasoning Errors and Biases in Generative AI
      • Identify Hallucinations, Reasoning Errors and Biases in LLM Output

    Hands-on Exercise: Experiment with hallucinations in testing with GenAI
    Hands-on Exercise: Experiment with reasoning errors in testing with GenAI

    • Mitigation techniques of GenAI hallucinations, reasoning errors and biases in software test tasks
    • Mitigation of Non-Deterministic Behavior of LLMs

• • Data Privacy and Security Risks of Generative AI in Software Testing
    • • Data Privacy and Security Risks Associated with Using Generative AI
      • Data Privacy and Vulnerabilities in Generative AI for Test processes and Tools
      • Mitigation Strategies to Protect Data Privacy and Enhance Security in Testing with Generative AI

    Hands-on Exercise: Recognize data privacy and security risks in a given Generative AI for testing case study

• • Energy Consumption and Environmental Impact of Generative AI in Software Testing
    • • The Impact of Using GenAI on Energy Consumption and CO2 Emissions

    Hands-on Exercise: Use a simulator to calculate the energy and CO? emissions for given test tasks with Generative AI

• • AI Regulations, Standards, and Best Practice Frameworks
    • AI Regulations, Standards and Frameworks Relevant to GenAI in Software Testing

LLM-Powered Test Infrastructure for Software Testing

• • Architectural Approaches for LLM-Powered Test Infrastructure
    • • Key Architectural Components and Concepts of LLM-Powered Test Infrastructure
      • Retrieval-Augmented Generation

    Hands-on Exercise: Experiment with Retrieval-Augmented Generation for a given test task

    • • The Role of LLM-Powered Agents in Automating Test processes

    Hands-on Exercise: Observe how an LLM-powered agent assists in automating a repetitive test task

• Fine-Tuning and LLMOps: Operationalizing Generative AI for Software Testing
  • • Fine-Tuning LLMs for Test tasks

  Hands-on Exercise: Observe an example of a fine-tuning process for a given test task and language model

  • LLMOps when Deploying and Managing LLMs for Software Testing

Deploying and Integrating Generative AI in Test organizations

• • Roadmap for the Adoption of Generative AI in Software Testing
    • • Risks of Shadow AI
      • Key Aspects of a Generative AI Strategy in Software Testing
      • Selecting LLMs/SLMs for Software Test Tasks

    Hands-on Exercise: Estimate the recurring costs of using Generative AI for a given test task

    • Phases when Adopting Generative AI in Software Testing

• Manage Change when Adopting Generative AI for Software Testing
  • Essential Skills and Knowledge for Testing with Generative AI
  • Building Generative AI Capabilities in Test Teams
  • Evolving Test Processes in AI-Enabled Test organizations.

Prerequisites

You must have obtained an ISTQB Foundation Level Certification (CTFL) to be eligible for the AI Testing certification exam.