📘 Free AAIA Sample Questions
Which of the following key performance indicators (KPIs) are MOST important when evaluating whether an AI
model meets business objectives?
A
Cost of resources required for AI model training
B
AI model accuracy in predicting actual outcomes
C
Frequency of AI model retraining
D
Number of users interacting with the AI model
Correct Answer:
B. AI model accuracy in predicting actual outcomes
Explanation:
The correct answer is B: AI model accuracy in predicting actual outcomes. Here's why:
The primary goal of deploying an AI model in a business context is to improve decision-making or automate
processes. This improvement hinges directly on the model's ability to accurately predict or classify data.
While other KPIs are important, accuracy directly reflects the value and effectiveness of the model in
achieving its intended purpose. A highly accurate model leads to better business outcomes, while a low
accuracy model can lead to incorrect decisions, inefficiencies, or even losses.
Option A, "Cost of resources required for AI model training," is crucial for budget management and ROI
calculation, especially in a cloud computing environment where resource consumption directly translates to
expenses. However, low training costs are meaningless if the model lacks accuracy.
Option C, "Frequency of AI model retraining," is related to model maintenance and preventing model drift
(where performance degrades over time due to changes in the underlying data). Retraining frequency is
important, but frequent retraining is only valuable if it results in improved accuracy. An accurate model that
requires infrequent retraining is better than a constantly retrained, inaccurate model.
Further research:
Option D, "Number of users interacting with the AI model," reflects adoption and usage, not necessarily the
model's performance or its contribution to business objectives. High usage of an inaccurate model can
amplify negative consequences. Usage is beneficial only if the model delivers correct or valuable results.
Therefore, accuracy is the most fundamental KPI because it directly addresses the core purpose of the AI
model: providing reliable predictions or classifications that support business goals. An accurate model is the
foundation upon which other factors, like cost-effectiveness, maintenance schedule, and user adoption, can
contribute to overall business success. Without accuracy, the other factors are largely irrelevant.
Google AI Principles: Outlines the importance of accuracy in responsible AI development.
(https://ai.google/principles/)
"Evaluating Machine Learning Models" - Google Developers: Explains various accuracy metrics and their
importance. (https://developers.google.com/machine-learning/crash-course/classification/accuracy)
NIST AI Risk Management Framework: Emphasizes the need to evaluate the performance, including
accuracy, of AI systems. (https://www.nist.gov/itl/ai-risk-management-framework)
The BEST way to prevent sensitive information disclosure by large language model (LLM) chatbots is through:
A
manual monitoring.
B
data sanitization.
C
data masking.
D
access controls.
Correct Answer:
B. data sanitization.
Explanation:
The correct answer is B. data sanitization. Here's why:
Large language model (LLM) chatbots are trained on massive datasets. These datasets can inadvertently
contain sensitive information like Personally Identifiable Information (PII), confidential business data, or even
proprietary code. If this sensitive data is incorporated into the LLM's training, the chatbot might inadvertently
regurgitate it in response to user queries.
Data sanitization involves the process of removing or obscuring sensitive information from the training data
before it's used to train the LLM. This proactive measure significantly reduces the risk of the chatbot leaking
this information. Techniques include redacting specific data points, generalizing data (e.g., replacing specific
addresses with city-level information), and employing differential privacy techniques during data generation.
While manual monitoring (A) can help detect instances of data leakage, it is reactive and doesn't prevent the
disclosure from happening in the first place. It's also incredibly resource-intensive and not scalable. Data
masking (C), while similar to sanitization, is more often associated with dynamic data protection, particularly in
databases, and may not be applied as comprehensively during LLM training data preparation. Access controls
(D) restrict who can use the chatbot and access its outputs, but they don't prevent the chatbot itself from
revealing sensitive information if it's been trained on that data. Think of it as locking the doors of a building
when the foundation (the LLM) is built on compromised material. Sanitizing the data during the build prevents
that compromise. Therefore, data sanitization is the most effective preventative measure to mitigate the risk
of sensitive information disclosure by LLM chatbots.
Supporting Links:
OWASP Al Security and Privacy Guide: https://owasp.org/www-project-ai-security-and-privacy-guide/ - This
resource provides guidance on securing Al systems, including data sanitization techniques.
NIST Al Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework - NIST
addresses issues related to Al safety, including data govemance and risk management related to sensitive
data.
Google Cloud Blog - Secure Al Framework: https://cloud.google.com/blog/products/ai-machine-
learning/secure-ai-framework-security-and-trust-in-generative-ai - Discusses the Secure Al Framework and
how data governance plays a crucial role.
An organization is using information gathered from customer accounts to train its Al chatbot. Which of the
following is the GREATEST risk associated with this practice?
A
Transparency
B
Al model hallucinations
C
Al bias
D
Disclosure of personal information
Correct Answer:
D. Disclosure of personal information
Explanation:
The correct answer is D, Disclosure of personal information, because using customer account data directly to
train an Al chatbot introduces the highest risk of inadvertently exposing Personally Identifiable Information
(PII).
Here's why:
While Al bias (C) is a significant concern when training Al models with potentially skewed data, the immediate
and most direct risk from using customer account data is the potential exposure of that data.
Transparency (A) refers to the explainability of the Al model's decisions and is not the immediate risk
presented by using customer data. Hallucinations (B) are when an Al model generates nonsensical or factually
incorrect outputs. While concerning, they aren't as directly linked to the specific scenario as data exposure.
If the training process isn't meticulously anonymized and de-identified, the chatbot could learn to associate
specific information with individual customers, leading to the unintentional disclosure of sensitive data during
interactions. Even seemingly innocuous details, when combined, can deanonymize individuals. Data breaches
are a significant legal and reputational risk.
Data privacy regulations like GDPR and CCPA impose stringent requirements on how personal data is
processed and protected, including its use in Al training. Failure to comply can result in hefty fines and
reputational damage. Using customer data without proper safeguards directly violates these principles.
The other options, while important, are secondary to the immediate and potentially devastating risk of
disclosing personal information. Protecting customer data is paramount, especially when utilizing Al,
necessitating careful planning, implementation, and ongoing monitoring. Effective anonymization techniques,
data minimization, and robust access controls are crucial.
Therefore, direct use of customer accounts to train a chatbot without careful processing, represents a higher
risk of personal data disclosure compared to bias, hallucinations or transparency.
Supporting Links:
1. NIST Al Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
2. GDPR - Official EU Website: https://gdpr.eu/
3. CCPA - California Consumer Privacy Act: https://oag.ca.gov/privacy/ccpa
An IS auditor is evaluating an organization’s incident management program to ensure it is sufficiently prepared to
manage AI-related incidents. Which of the following is MOST important for the auditor to validate?
A
The program includes processes to respond to AI model drift and data integrity attacks.
B
The program prioritizes incidents based on alignment with industry leading practices.
C
The program uses past AI-related incidents and resolutions to categorize current incidents.
D
The program mandates retraining AI systems after incidents are investigated.
Correct Answer:
A. The program includes processes to respond to AI model drift and data integrity attacks.
Explanation:
The most important aspect of an incident management program for AI is addressing AI-specific risks. Option A
directly addresses the unique challenges posed by AI systems, specifically model drift and data integrity
attacks, which are critical to maintaining AI's reliability and security. Model drift, where an AI's performance
degrades over time due to changes in input data, necessitates proactive monitoring and response
mechanisms. Data integrity attacks, which can compromise the AI's training data or input data leading to
incorrect or malicious outputs, also require robust detection and mitigation strategies.
While options B, C, and D represent valuable elements of a well-rounded incident management program, they
are secondary to the AI-specific considerations highlighted in option A. Option B focuses on aligning with
industry best practices, which is important but doesn't guarantee preparedness for unique AI risks. Option C,
using past incidents for categorization, is useful for general incident management, but might not address
novel AI attacks or drift patterns. Option D, mandating retraining, may be necessary in some cases, but it is
not universally applicable and does not cover all potential AI incidents.
In the context of AI systems, proactive management of model drift and data integrity is crucial. Ignoring these
vulnerabilities could lead to inaccurate predictions, biased outcomes, or even system manipulation, with
potentially severe consequences. Therefore, verifying that the incident management program effectively
addresses these risks is the most vital step in an AI audit. Focusing on AI-specific vulnerabilities is more
important than generic incident management practices when auditing AI systems.
Authoritative Links:
NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework (Provides a
framework for managing risks associated with AI systems, including model drift and data integrity.)
OWASP Top Ten AI: https://owasp.org/www-project-top-ten-for-large-language-model-applications/
(Addresses security vulnerabilities in AI applications, including data poisoning and model security.)
Which of the following is MOST important for an IS auditor to consider when identifying AI risk in a know your
customer (KYC) application within a banking organization?
A
Business disruption and financial impact
B
Intellectual property leakage and invalidation
C
Benchmarking against peer organizations
D
Incident response plan
Correct Answer:
A. Business disruption and financial impact
Explanation:
A. Business disruption and financial impact is the most critical consideration for an IS auditor when identifying
Al risk in a KYC application within a banking organization.
In the context of KYC compliance, Al systems analyze vast amounts of customer data to detect fraud and
ensure regulatory adherence. A failure in these systems can result in substantial business disruptions,
regulatory penalties, and financial liabilities. Such impacts can compromise the organization's operational
integrity and trust, essential elements in the banking sector's risk management landscape. Moreover, the
repercussions of inadequate Al risk assessments can extend beyond immediate financial losses, inciting
reputational damage that alters customer perceptions and diminishes market competitiveness.
Critical Evaluation of Other Options:
B. Intellectual property leakage and invalidation are significant but secondary in KYC applications. While
proprietary algorithms and data management strategies are vital, the immediate implications of systemic
failures on operational continuity and compliance take precedence. Intellectual property concerns become
tangential when foundational business operations are at risk.
C. Benchmarking against peer organizations is important for understanding industry standards but does not
address the unique risks associated with an organization's specific Al implementation in KYC. Reliance on peer
comparisons may lead to complacency and oversight of specific vulnerabilities inherent to the operational
context of the institution.
D. An incident response plan is essential for mitigating risks post-incident, but without identifying and
addressing risks proactively, the organization remains exposed to catastrophic failures. An effective audit
must prioritize proactive risk identification, acknowledging that an incident response plan is reactive by
nature.
For robust risk management strategies in Al applications:
https://www.isaca.org/resources/news-and-trends/newsletters/insights/2020/volume-6/understanding-ai-
governance-and-risk-management
https://www.ibm.com/blogs/research/2021/01/ai-governance-in-the-financial-industry/
https://www.pwc.com/gx/en/services/governance-risk-compliance/ai-risk-and-regulation-in-financial-
services.html
A digital bank utilizes an AI system to generate credit scores. Which of the following would BEST mitigate the risk
of sudden and unexplained changes in a borrower's credit score?
A
Using only data from the last six months to one year to avoid outdated information affecting the credit score
B
Ensuring the system is periodically reviewed and calibrated by human experts to maintain stability in
predictions
C
Obtaining and validating the credit scores from third-party agencies to cross-check AI-generated results
D
Allowing the AI to operate fully autonomously to prevent processing delays
Correct Answer:
C. Obtaining and validating the credit scores from third-party agencies to cross-check AI-generated results
Explanation:
Here's a detailed justification for why option C is the best choice to mitigate the risk of sudden and
unexplained changes in an Al-driven credit scoring system at a digital bank:
Option C, "Obtaining and validating the credit scores from third-party agencies to cross-check Al-generated
results," is the strongest mitigation because it provides an independent benchmark. Sudden and unexplained
changes in Al output often stem from data drift, model decay, or unexpected biases. Cross-checking with
established, independent credit scores from reputable agencies acts as a critical control to identify these
anomalies. If the Al-generated score diverges significantly from the third-party score, it signals a potential
problem within the Al system that needs investigation and correction.
Option A, while seemingly addressing outdated data, isn't comprehensive. It only deals with the age of data,
not the validity or potential bias in the training data. It wouldn't catch issues introduced by newly ingested but
flawed data.
Option B, periodic review and calibration, is a good practice but is reactive. Human experts may not catch
subtle shifts immediately. Independent validation offers real-time monitoring.
Option D, autonomous operation, exacerbates the problem. Removing human oversight increases the
likelihood of erroneous or biased scores going undetected, potentially causing significant financial harm and
reputational damage.
Therefore, comparing Al-generated scores to independent sources provides a crucial layer of validation,
ensuring greater reliability and stability in the credit scoring process. It is a robust control that directly
addresses the risk of sudden and unexplained score changes, offering timely alerts for potential issues in the
Al model's performance.
For further research on Al model validation and monitoring:
NIST Al Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework - Provides a
comprehensive framework for managing Al risks, including validation and monitoring.
MITRE ATLAS: https://atlas.mitre.org/ - ATLAS is a framework for adversarial threat landscape for artificial-
intelligence systems.
European Union's Al Act: https://artificialintelligenceact.eu/- This law set rules related to Al implementations
in the EU.
In the context of digital banking and Al, ensuring the reliability and fairness of Al credit scoring systems is
paramount. Independent validation is a key component of responsible Al implementation.
Which of the following is MOST important for an IS auditor to review during an AI system audit in order to
determine compliance with intellectual property and data rights?
A
Data performance metrics
B
Use of open-source intellectual property
C
Model runtime efficiency logs
D
Data usage agreements
Correct Answer:
D. Data usage agreements
Explanation:
The correct answer is D: Data usage agreements. Here's why:
During an AI system audit concerning intellectual property and data rights, the most important area for an IS
auditor to review is data usage agreements. These agreements are legally binding contracts that specify how
data can be used, processed, and shared. They outline the permitted uses, restrictions, and ownership rights
associated with the data used to train, validate, and operate the AI system. Confirming adherence to these
agreements is crucial for demonstrating compliance with intellectual property laws and preventing
unauthorized use or disclosure of sensitive information.
While open-source intellectual property (option B) is relevant, it's not the most important because data usage
agreements encompass a broader range of data sources, including proprietary and third-party data, which
may have stricter usage terms than open-source licenses. Data performance metrics (option A) and model
runtime efficiency logs (option C) primarily relate to the technical functionality and efficiency of the AI
system, not the legal and ethical aspects of data usage. Breaching a data usage agreement can result in
significant legal and financial consequences, making their review paramount in an AI audit focusing on IP and
data rights. Analyzing these agreements helps determine if data is being used within the defined legal and
ethical boundaries, thus mitigating risks related to intellectual property infringement and data privacy
violations. The auditor must verify that the AI system's data handling practices align with the stipulations of
these agreements to ensure responsible and lawful AI deployment.
Further Reading:
NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
GDPR (General Data Protection Regulation): https://gdpr-info.eu/
Which of the following is the PRIMARY objective of AI governance?
A
Implementing compliance and ethics controls for AI initiatives
B
Promoting a positive return on investment (ROI) from AI projects
C
Defining clear roles and responsibilities for AI development, use, and oversight
D
Ensuring controls over AI are designed well and operate effectively
Correct Answer:
C. Defining clear roles and responsibilities for AI development, use, and oversight
Explanation:
The primary objective of AI governance is C. Defining clear roles and responsibilities for AI development,
use, and oversight.
Here's why: AI governance establishes a framework for managing the risks and benefits associated with AI.
Before addressing compliance, ROI, or control effectiveness (all important components of governance), the
foundation must be laid by clearly defining who is accountable for what. This includes identifying roles for
data scientists, AI engineers, business owners, risk managers, and legal/compliance teams. Without clarity in
responsibilities, it's impossible to enforce ethical guidelines, ensure compliance with regulations, or even
measure the return on investment effectively.
Option A, implementing compliance and ethics controls, is a result of effective governance, not the primary
objective itself. Compliance and ethics stem from defined roles and policies. Option B, promoting ROI, is a
business objective, but secondary to the foundational governance that ensures sustainable and responsible AI
deployment. While desirable, ROI is not the central driving force behind establishing a robust AI governance
framework. Option D, ensuring controls over AI are well-designed and operate effectively, is again a
consequence of having clearly defined roles and responsibilities. Good controls cannot be implemented if
ownership and accountability are absent.
Establishing clear roles ensures appropriate oversight and accountability throughout the AI lifecycle,
including development, deployment, and maintenance. This clarity forms the basis for risk management,
ethical considerations, and compliance adherence. A well-defined governance structure enables
organizations to manage AI risks proactively, promoting trust and responsible AI adoption. This also allows for
better resource allocation, process standardization, and consistent decision-making concerning AI initiatives.
In essence, clearly defined roles and responsibilities are the cornerstone of effective AI governance, enabling
all other aspects of a comprehensive AI program.
Authoritative links for further research:
NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
OECD AI Principles: https://www.oecd.org/science/artificial-intelligence/recommendation-of-the-council-on
artificial-intelligence-C-2019-1118.htm
A healthcare organization uses patient data to train an AI model for early disease detection. Which of the following
practices provides the BEST assurance that personal data is secure and its integrity is maintained?
A
Implementing strict data access controls and conducting security tests
B
Encrypting stored data to reduce exposure and log access
C
Updating the AI model with new data and tracking changes
D
Anonymizing patient data and performing regular quality checks
Correct Answer:
B. Encrypting stored data to reduce exposure and log access
Explanation:
The best answer is B because it directly addresses both data security and integrity, key considerations when
dealing with sensitive personal data like patient information in AI applications. Encrypting stored data
protects the data from unauthorized access, rendering it unreadable even if a breach occurs. Data access logs
provide an audit trail, enabling monitoring and investigation of any suspicious activity. This helps maintain
data integrity by ensuring accountability and detect unauthorized modifications.
Option A, while important, focuses more on preventing unauthorized access but doesn't directly address data
integrity in case of a breach. Option C is about model performance and evolution, not data security and
integrity. Option D, anonymization, is crucial for privacy but, by itself, doesn't guarantee data integrity.
Anonymized data can still be corrupted or manipulated. Encryption and access logs offer robust, layered
protection. Encryption meets compliance requirements like HIPAA which mandate specific security measures
for protecting patient data.
data security and maintain its integrity in this AI application.
For further research:
Therefore, encrypting stored data and logging access is the most comprehensive approach to ensure patient
NIST Guidelines on Encryption and Key Management: https://csrc.nist.gov/projects/cryptographic
standards-and-guidelines
HIPAA Security Rule: https://www.hhs.gov/hipaa/for-professionals/security/index.html
Data Encryption Standard (DES): (Although outdated, provides foundational knowledge)
https://www.nist.gov/itl/applied-cybersecurity/resources/data-encryption-standard-des
An organization deploys an AI recruitment platform to screen job applicants. The IS auditor identifies that the
platform’s decisions may be influenced by model bias. Which of the following risk mitigation strategies is BEST for
the auditor to recommend?
A
Implement a process to periodically test the AI system for biases and adjust parameters as needed.
B
Suspend the use of the AI system until the training data can be verified for fairness and compliance.
C
Require manual reviews of all AI-generated recruitment decisions before hiring is finalized.
D
Retrain the AI model using an external data set certified for inclusivity and fairness.
Correct Answer:
A. Implement a process to periodically test the AI system for biases and adjust parameters as needed.
Explanation:
The best risk mitigation strategy in this scenario is A. Implement a process to periodically test the Al system
for biases and adjust parameters as needed.
Here's why:
Iterative Improvement: Al systems, especially those dealing with sensitive data like recruitment, require
continuous monitoring and improvement. Bias detection and mitigation isn't a one-time fix. Regular testing
allows for the identification of emerging biases due to data drift or model evolution.
Practicality: While ideal, suspending the system (B) might not be feasible due to business needs and
operational disruptions. Similarly, retraining with a certified dataset (D) can be costly and might not
completely eliminate bias as bias can exist in the training process itself. Manual reviews of all decisions (C)
would undermine the efficiency gains expected from Al deployment and would require significant resources.
Bias Detection Methods: Implementing a testing process means utilizing various techniques to check for bias.
These can include:
Statistical parity: Ensuring equal selection rates across protected groups.
Equal opportunity: Ensuring equal true positive rates across protected groups.
Predictive parity: Ensuring equal positive predictive values across protected groups.
Parameter Adjustment: The testing process should be followed by parameter adjustments. This could involve
re-weighting data, adjusting the model's objective function, or using bias mitigation algorithms.
Feasibility and Cost-Effectiveness: Option A balances the need for bias mitigation with the practical
constraints of deploying and maintaining an Al system. It allows the organization to continue benefiting from
the Al platform while actively working to reduce bias.
Continuous Monitoring: Bias can creep back into the system over time as the data it processes evolves. Thus,
continuous monitoring and periodic testing are essential.
Further Research:
NIST Al Risk Management Framework: This framework provides guidance on identifying, assessing, and
managing risks related to Al systems, including bias. https://www.nist.gov/itl/ai-risk-management-framework
IBM Al Fairness 360: An open-source toolkit that includes algorithms, metrics, and explanations to help
detect and mitigate bias in machine learning models. https://aif360.mybluemix.net/
Questions: 1-10 out of 445
Continue Full Practice..
GET ALL 445 QUESTIONS