AI penetration testing: what should an assessment actually cover?
Compare test scopes across identities, retrieval, model behaviour and tools. Specify evidence, rules of engagement and retesting before buying an assessment.
Two proposals can both promise “AI penetration testing” while testing very different systems. One may submit adversarial prompts to a model endpoint. Another may examine the application’s identities, retrieval pipeline, tools and downstream effects. A finding count cannot make those scopes comparable.
Start with the business consequence you need to investigate: disclosure of another customer’s records, an unauthorised account change, an unsafe tool action or service disruption. Then identify every component that could make that consequence possible.
Ask for a boundary map before a payload list
For a customer-support assistant, map the user interface, session and tenant identity, retrieval service, model endpoint, tool gateway and ticketing API. Record where untrusted material enters and which identity each component uses. Include caches, conversation storage, file uploads and administrative routes when they are part of the deployed system.
A model refusing a request does not establish that the application denied access. Conversely, an objectionable model response is not automatically evidence of data theft or an executed transaction. Require the report to distinguish model behaviour from application authorisation and downstream side effects.
Put these scope decisions into the statement of work
| Layer | Question to resolve | Evidence to request |
|---|---|---|
| Application and API | Are authentication, tenant separation, object permissions and uploads included? | Identity-specific requests and server-side decisions. |
| Retrieval and context | Can another user’s data reach model context, cached responses or citations? | Document provenance and authorised context traces. |
| Model interaction | Which direct and indirect injection paths are in scope? | Input, configuration, repetitions and observed outputs. |
| Tools and actions | Can manipulated output cross an action or approval boundary? | Tool arguments, execution identity and downstream effect. |
| Operations | Are rate limits, telemetry and recovery included? | Agreed bounded tests, detection signals and recovery observations. |
OWASP’s Excessive Agency guidance identifies risks from unnecessary functionality, excessive permissions and excessive autonomy. Use that taxonomy to ask concrete questions about the deployed tool set. It is not a substitute for checking which tools are actually reachable or what credentials they hold.
Define what counts as a demonstrated finding
Consider a proposed test in which an external support article contains instructions to redirect a ticket export. The assessment should first show that the normal support task works with a clean article. It should then record whether the altered article changes the model’s proposal, whether the tool gateway permits the new destination and whether any export occurs. Use a controlled destination and synthetic records.
Report the highest boundary actually crossed. “The model proposed a different destination” and “the system delivered protected records to it” are different outcomes. If execution is safely blocked, retain that positive control evidence. This scenario is an assessment design, not a MicroApz customer result.
For stochastic behaviours, record attempts and conditions: for example, an illustrative result of 3 successes in 20 attempts under one configuration would describe those trials only. It would not establish a universal exploit probability. Preserve model identifier, settings, system instructions where authorised, tool versions and test date so a retest has meaning.
Agree the rules of engagement
Specify allowed tenants, accounts, environments, time windows, test data and downstream destinations. Set rate and cost limits. Identify destructive actions that must be simulated, the stop condition and the person who can halt testing. Obtain explicit permission for third-party systems; access to your application does not automatically authorise testing its provider.
Document exclusions such as model training infrastructure, provider internals, physical systems, availability attacks or business workflows unavailable in the test environment. An assessment can be useful with exclusions, but its conclusion must stay within them.
Request a report you can act on and retest
Each finding should contain prerequisites, affected component and version, reproduction steps, observed impact, evidence, root cause, proposed remediation and a retest condition. Include unsuccessful tests and coverage gaps in a separate coverage record so “no findings” cannot be mistaken for “everything tested”.
Ask which fixes belong in deterministic application controls and which reduce model-level risk. Specify whether retesting is included, when it occurs and how changes to the deployment affect the result. An AI penetration test is a scoped security assessment; it does not by itself establish legal compliance, complete AI trustworthiness or certification.
Download the assessment worksheet (CSV) and use the AI assessment rows to record scope, expected behaviour and actual evidence. All result fields are intentionally blank.
Source basis
OWASP AI Testing Guide addresses broader AI trustworthiness testing. OWASP Excessive Agency supplies the tool-risk taxonomy used above. The boundary map, procurement questions and example reporting criteria are MicroApz’s proposed assessment approach; they are not a claim of OWASP certification or endorsement.