How to choose an AI security lab

Step 1: define the target

Providers specialise by target. Sending an application brief to a frontier evaluation lab, or a model capability question to a guardrail vendor, wastes time on both sides.

Step 2: write the threat model

A threat model is a short document that says who might attack the system, what they want and what would hurt you most. It keeps the testers focused on realistic harm instead of curiosities.

• Attackers. Anonymous internet users, logged-in customers, insiders, or third parties whose content the system reads.
• Assets. Customer data, internal documents, credentials, money movement, the ability to send messages in your name.
• Entry points. The chat box, uploaded files, emails, web pages, API calls, connected tools.
• Worst outcomes. Rank three to five outcomes you most need to rule out, and say which are legal or regulatory matters.
• Out of scope. State what must not be touched, such as production data or third-party systems.

Step 3: decide what you need to receive

Agree the deliverables in writing. A list of clever jailbreaks is of little use if nobody can reproduce or prioritise them.

• A technical report with each finding, its severity, evidence and exact reproduction steps.
• A summary for management that states residual risk in plain terms.
• Remediation advice specific to your architecture, not generic guidance.
• The attack prompts and scripts in a reusable form, so you can turn them into regression tests.
• A retest after fixes, with an updated report.
• If required, a letter or summary suitable for customers, auditors or regulators.

Step 4: settle disclosure and data handling

Disclosure is a larger issue in AI security than in ordinary testing. Research labs and non-profits often want to publish methods or results. Competition platforms expose your system to many outside participants. A weakness found in your application may in fact sit in the underlying model, which means the model vendor should be told.

Before work starts, agree who may publish what and when, how findings in third-party models are reported to their vendors, and how long the lab keeps your prompts, logs and data. If the lab has been acquired by a larger vendor, ask whether your findings can be used to improve the parent company's products.

Questions that separate strong labs from weak ones

• Can you walk us through your method on a system like ours?
• Can your team fix the findings in our code, and is a retest included?
• What share of this engagement is manual work by named testers?
• How do you test indirect prompt injection through documents, emails and web content?
• How do you rate severity when an attack succeeds only some of the time?
• Which parts of our system will remain untested at the end?
• Who owns your company today, and has that changed in the past two years?

Independence and conflicts of interest

Ask how the lab is funded and what else it sells. A vendor that finds problems and also sells the product that blocks them has an incentive to frame findings around its own tool. A lab funded by model developers may be cautious about criticising them. Both situations should be known before you sign.

A team that sells its time and its engineering, as BlackCard does, has a simple incentive: find the real weaknesses, fix them and show in a retest that the fixes hold. Its findings are ranked by the risk to your system.