---
title: "AI Security in 2026: The Scope, the Boundaries and the Order to Work In"
description: "The term covers two distinct jobs: defending the model system and bounding what it can reach. Here is the scope, the structural differences, and a sequenced first month."
author: "Mara Voss"
category: "Defensive Architecture & Security Controls"
date: 2026-10-13T13:05:07.692Z
canonical: "https://agenticcyber.co/blog/ai-security-what-it-covers-in-2026-and-where-to-start"
---

# AI Security in 2026: The Scope, the Boundaries and the Order to Work In

![cctv surveillance camera, cctv, security](https://cdn.pixabay.com/photo/2022/06/17/09/50/cctv-surveillance-camera-7267551_1280.jpg?w=1200&q=75)

> The term covers two distinct jobs: defending the model system and bounding what it can reach. Here is the scope, the structural differences, and a sequenced first month.

Five teams, five definitions. One means model weights walking out of the building. One means a support bot being argued into saying something embarrassing. One means customer records sitting in a prompt log. One means a procurement questionnaire. One means a cloned voice on a call with finance. All five are real problems. Funding them as a single budget line is how the money ends up on the layer that was never going to fail.

So here is a working map of the field as it stands in October 2026: what sits inside the term, what makes these failures structurally unlike the ones application security already handles, and what a small team does first, second and third. It is written for the engineers shipping the features and for the reviewers who have to sign them off.

## Two Jobs Wearing One Name

Pull the label apart and there are two separate pieces of work.

- **Defending the machinery.** The model, the data it learned from, the system prompts, the retrieval store, the tool definitions, the plugins, and the build-and-serve pipeline behind all of it.

- **Bounding the consequences.** The credentials, the records and the operations reachable by a system that a model steers, once that system is wired into something real. This half expands every time an engineer adds a tool, and it expands without anyone touching the model.

Shared vocabulary helps more than it sounds like it should. NIST published its adversarial machine learning taxonomy, AI 100-2 E2025, in March 2025, and gives its purpose as "establishing a common language for the rapidly developing AML landscape" in the abstract of [AI 100-2 E2025](https://csrc.nist.gov/pubs/ai/100/2/e2025/final). The document sorts attacks by the lifecycle stage they strike and by what an adversary knows and can do, and it names four broad classes: poisoning, evasion, abuse, and breach of privacy. Its practical value is that it keeps "someone corrupted what the model learned" in a different box from "someone controls what the model is reading right now". Those are different investigations with different owners and different fixes.

## What Actually Changed

Three structural properties explain why the familiar playbook does not transfer cleanly.

**Instruction and data arrive on the same wire.** A conventional application keeps code and input in separate lanes by construction. A model takes both as one token stream and has no dependable way to tell them apart. An invoice, a scraped page, a calendar invitation or a commit message can therefore carry something the system obeys. That property is why prompt injection leads the [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) as LLM01.

**Behaviour is sampled, not computed.** You cannot enumerate the outputs and test them one at a time, so any control whose correctness depends on the model choosing sensibly holds right up until the input nobody imagined.

**Reach outruns review.** A question-answering assistant has a small radius. Attach a mail tool, a ticketing API and a database connector and the radius becomes the union of those systems. OWASP files this as LLM06 and attributes it to three causes, too much functionality, too much permission and too much autonomy, on its [excessive agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) page.

The rest of that list reads as decent terrain mapping: disclosure of sensitive information, supply chain, poisoning of training data and models, mishandled output, system prompt leakage, weaknesses in vectors and embeddings, misinformation and unbounded consumption. Read it as a map of where trouble lives, not as a work queue. Coverage matters less than order.

## First: Write Down What You Already Have

Nothing downstream applies to a system you cannot name. Enumerate the model endpoints, the agents, the tools each agent may call, the data each one reads, and the human who owns it. Include the pilot nobody decommissioned, the agent someone in marketing built inside a SaaS product, and the key that a notebook on a laptop still uses. This is ordinary attack surface work aimed at a new class of asset, and [inventory before controls](/attack-surface-management-for-ai-agents-inventory-before-controls) covers how to assemble the list and keep it from going stale.

## Second: Shrink the Reach Before You Shop

The highest-value control at this stage carries no licence fee. Issue each agent a principal of its own rather than a key shared with a service. Remove the tools it was granted for a use case that never shipped. Swap long-lived API keys for narrow credentials with a short life. When an injected instruction can reach nothing but low-value, read-only data, you have converted a breach into a log line. The traps specific to non-human identities are collected in [least privilege for AI agents](/least-privilege-harder-with-ai-agents).

## Third: Treat Generated Text as Hostile Until Checked

Whatever the model emits may have been shaped by whoever influenced what it read, so it earns the same validation you already apply to a form field. Check it before it becomes a shell command, a database query, a rendered page or a request to a neighbouring service. OWASP tracks this separately as LLM05 precisely because teams who defend the input still pipe the output straight into an interpreter.

## Fourth: Put a Person Where Undo Does Not Exist

Write the short list of actions that have no reverse: moving funds, deleting records, messaging somebody outside the company, changing a permission. Require a human decision on exactly those, and show that human the real parameters rather than the model's account of them. Keep the list short. An approval queue that fires forty times a day trains its reviewer to click.

## Fifth: Log Enough to Rebuild a Run

When an agent behaves oddly, the questions are always the same four: who started the task, what text the model had in front of it, which tool executed, and with which arguments. If one of the four is missing, the investigation ends in speculation. What responders and auditors tend to ask for is itemised in [audit log evidence requirements](/agentic-ai-audit-log-evidence-requirements).

## Sixth: Keep an Adversarial Suite in Version Control

Write the attacks you expect, run them before release, and run them again after every change to a prompt, a model version or a tool definition. Holding the cases in the repository turns a regression into a failed check instead of a customer report. The aim is not a score. It is to find out where a control gives way while the environment still belongs to you.

## Knowing Which Layer a Control Belongs To

Teams lose weeks arguing about a control that was never going to live where they were putting it. Input filtering sits at the prompt layer and cannot bind an action. A tool-call allowlist sits at the execution layer and knows nothing about what the model read. Egress rules sit in the network and do not care which agent asked. Sort your intended controls by layer before you buy or build any of them, and the duplicates and the gaps both become obvious. Our [controls catalog by layer](/ai-security-controls-catalog-by-layer) sets out the layers and what each one can honestly enforce.

## Hanging the Work on a Framework

Once more than one team is involved, the sequence above needs an ownership model around it. NIST released the voluntary [AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) on January 26, 2023, split across the functions it labels Govern, Map, Measure and Manage, and added a Generative AI Profile, NIST-AI-600-1, on July 26, 2024. Neither document writes your egress rule. They answer who owns which risk and how anyone will know it is being held, which is the part that fails silently in a program with no structure.

## The Documents That Moved This Year

The reading list changed under this field in the past twelve months, and two of the changes are worth acting on rather than noting.

There is now an agent-specific risk list. The OWASP GenAI Security Project published [the Top 10 for Agentic Applications for 2026](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications/) on 9 December 2025, covering the risks that come from autonomy, delegation, memory and agents talking to each other, with its numbered categories inside the downloadable document. If your systems call tools, that list is the one to read first and the LLM-application list becomes background. The two overlap, and they are not interchangeable: one is organised around a model in a request, the other around a process that acts.

Control guidance for agents is also in flight, with dates you can plan against. NIST's [Control Overlays for Securing AI Systems project](https://csrc.nist.gov/projects/cosais) is building overlays that apply the SP 800-53 catalogue to specific AI use cases, and two of the five named use cases are single-agent and multi-agent systems. Its concept paper is dated 14 August 2025 and an annotated outline for the predictive AI overlay followed on 8 January 2026, prepared for a workshop on 14 January 2026 with feedback requested by 13 February 2026. The overlays are planned as interagency reports rather than as a revision of the control catalogue, and no initial public draft had appeared when this was written. The practical move: record which of those five use cases each of your systems belongs to now, while the inventory is being built, so that mapping an overlay onto it later is a lookup instead of a project.

If you sell into Europe, one date on your tracker probably needs editing. Under the Digital Omnibus agreement, [summarised here by Gibson Dunn](https://www.gibsondunn.com/eu-ai-act-omnibus-agreement-postponed-high-risk-deadlines/), the obligations for stand-alone high-risk systems move from 2 August 2026 to 2 December 2027, and embedded high-risk AI moves from 2 August 2027 to 2 August 2028, binding only once the text is published in the Official Journal. That is runway, not relief.

On the sixth step above, there is a free place to start rather than a procurement cycle. [garak](https://github.com/NVIDIA/garak), maintained in NVIDIA's GitHub organisation under the Apache-2.0 licence, is a model vulnerability scanner installed with a single pip command, with probe families covering prompt injection, package hallucination, encoding tricks and training-data replay. Its documentation describes probing a model and claims no coverage of agents or tool use, so treat a clean run as evidence about one layer. It belongs in the repository beside your own cases, not instead of them.

## Read One Document, Not Four

Here is where small teams lose their first quarter, and the advice to "ground the programme in a framework" is what does it. Four documents get recommended together: an adversarial machine learning taxonomy, a risk management framework, a generative AI profile, a top ten list. Each is good. Read in sequence they produce a vocabulary, a committee and no closed gaps, because none of them tells you which tool your invoice agent should stop being allowed to call.

So the call, for an engineering team shipping agents: read the agent risk list, and nothing else, until the first three steps above are done. The taxonomy and the risk framework are written for the person who has to allocate ownership across several teams and defend the allocation to an assessor. That is a real job and usually not yours, and reading them early converts a week of work into a month of mapping exercises. When the agent overlays reach public draft they will matter for the same reason the control catalogue matters: they say what to implement. The taxonomy says what to call things.

If you own the policy rather than the code, or you are preparing for a regulated assessment, read them in the other order, because the ownership model is the deliverable and the egress rule is somebody else's ticket. The same holds if you train or fine-tune on proprietary data, where poisoning and extraction move from the background of the taxonomy into your actual threat model. For the common case, a team wiring a bought model into company systems, reach is where the incident comes from, and the reading list is not where the fix is.

## What the Map Does Not Decide for You

Two things get mistaken for AI security and sit elsewhere. Synthetic voice and video used against your staff is social engineering: it belongs with payment verification, callback procedures and awareness work, not with model hardening. And model behaviour that is unpleasant without an attacker, meaning the safety and quality agenda, is a product problem answered by evaluation rather than by a control. Both deserve owners. Neither should absorb the budget that the two jobs above need.

## Where This Guidance Runs Out

If the system touches regulated data, moves funds or acts on production infrastructure, get an independent review from a qualified security team before it goes live. An article is orientation, not an audit, and a compliance or legal question belongs with the people who carry that responsibility inside your organisation.

No step here removes risk. Each either lowers the chance of a bad day or reduces what a bad day costs. Begin with the inventory. Every later step in this order quietly assumes one already exists.

## FAQ

### Who should own this work in a company with no AI security team?

Split it the way the two jobs split. The platform or infrastructure team owns identity, credentials, egress and logging for agents. The team shipping the feature owns the tool definitions, the approval points and the adversarial test set. A named reviewer signs off before launch.

### How is this different from AI safety work?

Safety work asks whether the system behaves acceptably with no adversary present, and it is answered mainly with evaluation. Security work assumes somebody is trying, and it is answered with identity, isolation, validation and telemetry. The same incident report can involve both.

### Does any of this change if we only call a hosted model through an API?

The first job shrinks, because the provider holds the training data and the serving stack. The second job does not shrink at all. Prompt handling, tool permissions, output validation and logging stay yours no matter whose weights are answering.

### Where do deepfakes and cloned voices fit?

They are a social engineering problem rather than a model-hardening one. The controls that work are process controls: callback verification, dual approval on payment changes, and training that assumes a convincing voice proves nothing.


---
Source: https://agenticcyber.co/blog/ai-security-what-it-covers-in-2026-and-where-to-start