CT-AI Exam Dumps - Certified Tester AI Testing Exam

A/B testing is a statistical testing method that compares two different versions of a system to determine which one performs better. In this scenario, theold NLP algorithmwas rated for accuracy, and after the update, thenew algorithmwas also rated by users. A statistical test was performed to compare the two versions, which is the fundamental approach ofA/B testing.

A/B testing is commonly used in:

User experience testing(e.g., comparing different versions of a website).

ML model evaluation(e.g., comparing two AI-based classifiers).

Performance assessment(e.g., determining if a new recommendation algorithm is more effective).

This approach allows for data-driven decisions, ensuring that any changes to the system result in meaningful improvements.

Section 9.4 - A/B Testingstates that A/B testing is used to compare updates in AI-based systems to determine if the newer version is better​.

Reference from ISTQB Certified Tester AI Testing Study Guide:

The problem described in the question is a classic case ofconcept drift. Concept drift occurs when the relationship between input variables and the output variable changes over time, leading to a decline in model accuracy​.

In this scenario, theaverage passenger and baggage weightsused in the model changed due to seasonal variations, but the model was not updated accordingly. This resulted in inaccurate predictions for fuel requirements in the winter season. This is an example ofseasonal drift, where model behavior changes periodically due to recurring trends (e.g., higher luggage weights in winter compared to summer).

To prevent such problems:

Themodel should be regularly testedfor concept drift against agreed ML functional performance criteria​.

Exploratory Data Analysis (EDA)should be performed periodically to detect gradual changes in input distributions.

Retraining of the modelwith updated training data should be done to maintain accuracy.

If drift is detected, mitigation techniques such asincremental learning, retraining with new data, or adjusting model parametersshould be employed.

Option B (Easing the performance standard instead of addressing drift): Lowering the performance standard is not a solution; it only masks the problem without fixing it. Instead, regular testing and retraining should be used to handle drift properly​.

Option C (Corruption and reloading the model): Model corruption is unrelated to this issue. Corruption refers to accidental or malicious damage to the model or data, whereas this case is due to a changing data environment​.

Option D (Lack of transparency): Transparency refers to how understandable the model’s decisions are, but the problem here is a change in data distributions, making drift the primary concern​.

ISTQB CT-AI Syllabus (Section 7.6: Testing for Concept Drift)

"The operational environment can change over time without the trained model changing correspondingly. This phenomenon is known as concept drift and typically causes the outputs of the model to become increasingly less accurate and less useful."​

"Systems that may be prone to concept drift should be regularly tested against their agreed ML functional performance criteria to ensure that any occurrences of concept drift are detected soon enough for the problem to be mitigated."​

ISTQB CT-AI Syllabus (Section 7.7: Selecting a Test Approach for an ML System)

"If concept drift is detected, it may be mitigated by retraining the system with up-to-date training data followed by confirmation testing, regression testing, and possibly A/B testing where the updated system must outperform the original system."​

Why Other Options Are Incorrect:Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:Since the question describes a situation whereseasonal variations affected input data distributions, the correct answer isA: The model suffers from drift and therefore should be regularly tested to ensure that any occurrences of drift are detected soon enough for the problem to be mitigated.

Understanding Supervised Learning - RegressionSupervised learning is a category of machine learning where the model is trained on labeled data. Within this category,regressionis used when the goal is to predict a continuous numeric value.

Regressiondeals with problems where the output variable is continuous in nature, meaning it can take any numerical value within a range.

Common examples include predicting prices, estimating demand, and analyzing production trends.

(A) Determining the optimal age for a chicken's egg-laying production using input data of the chicken's age and average daily egg production for one million chickens.✅(Correct)

This is a classicregression problembecause it involves predicting a continuous variable:daily egg productionbased on the input variablechicken's age.

The goal is to find a numerical relationship between age and egg production, which makesregression the appropriate supervised learning method.

(B) Recognizing a knife in carry-on luggage at a security checkpoint in an airport scanner.❌(Incorrect)

This is animage recognition task, which falls underclassification, not regression.

Classification problems involve assigning inputs to discrete categories (e.g., "knife detected" or "no knife detected").

(C) Determining if an animal is a pig or a cow based on image recognition.❌(Incorrect)

This is anotherclassification problemwhere the goal is to categorize an image into one of two labels (pig or cow).

(D) Predicting shopper purchasing behavior based on the category of shopper and the positioning of promotional displays within a store.❌(Incorrect)

This problem could involve a mix ofclassificationandassociation rule learning, but it does not explicitly predict a continuous variable in the way regression does.

Regression is used when predicting a numeric output."Predicting the age of a person based on input data about their habits or predicting the future prices of stocks are examples of problems that use regression."​

Supervised learning problems are divided into classification and regression."If the output is numeric and continuous in nature, it may be regression."​

Regression is commonly used for predicting numerical trends over time."Regression models result in a numerical or continuous output value for a given input."​

Analysis of Answer ChoicesReferences from ISTQB Certified Tester AI Testing Study GuideThus,option A is the correct answer, as it aligns with the principles of regression-based supervised learning.

Clustering is a type of machine learning technique used to group similar data points into clusters. It is a key concept in unsupervised learning, where the algorithm tries to find patterns or groupings in data without prior knowledge of output classes. Let's analyze each option:

A. Clustering is classification of a continuous quantity.

This is incorrect. Classification typically involves discrete categories, whereas clustering involves grouping similar data points. Classification of continuous quantities is generally referred to as regression.

B. Clustering is supervised learning.

This is incorrect. Clustering is an unsupervised learning technique because it does not rely on labeled data.

C. Clustering is done without prior knowledge of output classes.

This is correct. In clustering, the algorithm groups data points into clusters without any prior knowledge of the classes. It discovers the inherent structure in the data.

D. Clustering requires you to know the classes.

This is incorrect. Clustering does not require prior knowledge of classes. Instead, it aims to identify and form the classes or groups based on the data itself.

Therefore, the correct answer isCbecause clustering is an unsupervised learning technique done without prior knowledge of output classes​​.

Detecting biases in the ML pipeline involves various tests to ensure fairness and accuracy throughout the ML process.

Testing the distribution shift in the training data for inappropriate bias (A): This involves checking if there is any shift in the data distribution that could lead to bias in the model. It is an important test but not the most direct method for detecting biases.

Test the model during model evaluation for data bias (B): This is a critical stage where the model is evaluated to detect any biases in the data it was trained on. It directly addresses potential data biases in the model.

Testing the data pipeline for any sources for algorithmic bias (C): This test is crucial as it helps identify biases that may originate from the data processing and transformation stages within the pipeline. Detecting sources of algorithmic bias ensures that the model does not inherit biases from these processes.

Check the input test data for potential sample bias (D): While this is an important step, it focuses more on the input data and less on the overall data pipeline.

Hence, the most likely useful test to help detect different kinds of biases in the ML pipeline isB. Test the model during model evaluation for data bias.

References:

ISTQB CT-AI Syllabus Section 8.3 on Testing for Algorithmic, Sample, and Inappropriate Bias discusses various tests that can be performed to detect biases at different stages of the ML pipeline.

Sample Exam Questions document, Question #32 highlights the importance of evaluating the model for biases.