What an AI security lab does: red teaming, evaluations, audits
An AI security lab finds out how an AI system can be made to fail and how badly. It does this through three main activities: red teaming, evaluations and audits. Many labs add two product-style services, supply chain scanning and runtime protection. The terms are often used loosely, so this page defines each one and says what you receive at the end.
The five activities at a glance
| Activity | Question it answers | Typical output | Who offers it |
|---|---|---|---|
| Red teaming | Can a determined attacker make this system misbehave? | Report of successful attacks with reproduction steps | Applied teams such as BlackCard, frontier labs, product vendors |
| Model evaluation | How does this model score on a fixed set of risky tasks? | Scores that can be compared between versions | Frontier labs, non-profits, government institutes |
| Audit or assurance review | Is the whole system built and operated securely? | Findings report with severity ratings and fixes | Applied teams such as BlackCard, security consultancies |
| ML supply chain scanning | Are the model files and dependencies we download safe? | Scan results per artefact, often in the build pipeline | Product vendors |
| Runtime protection | Can attacks be blocked while the system is live? | A filtering or policy layer with logs and alerts | Product vendors |
Red teaming
Red teaming borrows its name from military and security practice. A team plays the adversary and tries to reach a harmful goal by any route the rules allow. Against a language model, typical goals are to bypass safety rules (jailbreaking), to extract the system prompt or private data, and to hijack an agent through text hidden in a document or web page (indirect prompt injection).
The work can be manual, automated or crowdsourced. Manual testers find creative, multi-step attacks. Automated tools generate thousands of attack variants and are suited to regression testing on every release. Public competitions add breadth by inviting many outside participants. Results describe what was tried and what worked. They cannot prove that no attack exists.
Model evaluations
An evaluation is a standard test. The lab builds a set of tasks, runs the model against them under controlled conditions and reports a score. In security work the tasks often measure dangerous capability: whether a model can find and exploit software vulnerabilities, carry out long tasks without supervision, or behave deceptively when it believes it is being observed.
Evaluations matter most to model developers and regulators, who use them to decide whether a model is safe to release and which safeguards it needs. Results appear in the system cards that developers publish with new models. For an enterprise buying a model, published evaluations are useful background, but they say little about the security of the application built on top.
Audits and assurance reviews
An audit looks beyond the model to everything around it. Reviewers examine the architecture, the training and deployment pipeline, access controls, the tools an agent may call, the handling of secrets and the way outputs are passed to other software. Much of this is classic security engineering applied to a new kind of system.
The deliverable is a written report with findings ranked by severity and recommended fixes, followed by a retest. BlackCard goes one step further: the team that writes the findings also makes the fixes in the code and retests each one.
Supply chain scanning and runtime protection
These services are delivered as software, not as projects. They do not replace testing. They reduce the chance that a known class of attack succeeds between tests.
- ML supply chain scanning checks model files, datasets and libraries before use. Some model file formats can carry executable code, so a downloaded model deserves the same suspicion as any other third-party software.
- Runtime protection sits between users, the model and its tools while the system is live. It inspects prompts, retrieved content and outputs, and blocks or flags what breaks policy.
- AI asset discovery inventories the models, agents and AI services in use across a company, including those adopted without approval.
What a lab does not do
No lab can certify that an AI system is secure. Language models respond to an unbounded range of inputs, and new attack techniques appear regularly. A good lab is explicit about this limit. It states what was in scope, how much effort was spent, which attacks were tried and what remains untested. Treat any promise of complete protection as a warning sign.
Where BlackCard fits
BlackCard works where most company AI risk sits: the application and the agent. It reviews the architecture, attacks the system with direct and indirect prompt injection, checks for data leaks and cuts agent permissions to what the task needs. The same senior team then fixes the findings in the code and retests them. Small and mid-size companies get one point of contact, in English or German, and a free 15-minute call to set the scope.
Common questions
Is AI red teaming the same as penetration testing?
They overlap but are not the same. Penetration testing targets software and infrastructure flaws with well-defined outcomes. AI red teaming also targets model behaviour, where results are probabilistic and an attack may work only some of the time.
Do all AI security labs offer all five activities?
No. Frontier labs concentrate on red teaming and evaluations of models. Product vendors concentrate on automated testing, scanning and runtime protection. Consultancies concentrate on audits. Applied teams such as BlackCard test the application and agent layer and fix what they find. Check each provider's stated scope before you compare quotes.