What Is AI Security? A Practical Guide to Securing AI Systems
Most organisations now have two AI security problems, not one. The first is familiar: attackers using AI to write more convincing phishing, generate malware faster, and fake voices and faces well enough to authorise a payment. The second is newer. The AI systems you build, buy, and embed in your products are themselves attack surfaces, and they behave in ways that traditional controls were never designed for.
The NCSC’s September 2026 guidance on shadow AI cites research finding that 71% of employees use AI tools their employer has not approved. At the same time, the UK’s official breach statistics show adoption clearly running ahead of assurance: 31% of UK businesses are using or considering AI, and only 24% of those have cyber security practices in place to manage AI risk (Cyber Security Breaches Survey 2025/2026, DSIT and Home Office, published 30 April 2026).
AI security is the practice of protecting AI systems and everything around them, the model, its training and retrieval data, the prompts and tools it accepts, and the organisation that depends on its output. In the UK, it now has a defined shape: the NCSC’s secure AI development guidelines, the DSIT code of practice, and a matching European standard.
This guide explains what AI security covers, where it differs from application and LLM security, the risks that actually bite, and a practical control set you can map onto a real programme.
Key Takeaways
- AI security covers two problems at once: securing the AI systems you run, and defending against attackers who use AI against you.
- Only 24% of UK businesses that are using or considering AI have cyber security practices to manage AI risk, against 43% that reported a breach in the last year.
- Prompt injection tops the OWASP Top 10 for LLM Applications 2025, followed by sensitive information disclosure, supply chain, and data and model poisoning.
- The UK Code of Practice for the Cyber Security of AI sets 13 voluntary principles across five lifecycle phases, and aligns to ETSI EN 304 223.
- Most of AI security is good security done deliberately: threat modelling, least privilege, supply chain visibility, monitoring, and an incident plan that accounts for models.
What is AI security?
AI security is the set of controls, processes, and governance that keep an AI system behaving as intended under adversarial pressure, and limit the damage when it does not. It covers the model itself, the data that trains and grounds it, the pipeline that builds and deploys it, the applications and agents that call it, and the people who use it.
That scope is wider than it sounds, because an AI system is rarely a single artefact. A typical deployment is a foundation model from a third party, a retrieval layer over your own documents, an orchestration framework, a set of tools or APIs the model can call, and a user interface. Each of those is a component with its own failure and attack modes, and each one inherits trust from the others.
AI security also has to account for the fact that, unlike a conventional application, an AI system does not have a fixed set of code paths. Its behaviour depends on input it has never seen, and its output is probabilistic. You cannot patch a prompt in the way you patch a query. That is why the controls that matter most are architectural, least privilege, constrained agency, careful data boundaries, rather than signature-based.
Finally, AI security includes the defensive use of AI: detection, triage and response tooling that leans on models to keep up with machine-speed attacks.
AI security vs cyber security vs LLM security
These terms overlap, and vendors often use them loosely. The useful distinction is scope.
Cyber security is the parent discipline. It protects systems, networks, identities and data, and AI security is now one of its domains.
AI security covers any AI system: classical machine learning models, computer vision, recommendation engines, forecasting systems, generative models and the agents built on them. It is about the whole lifecycle, including training, supply chain, and end of life.
LLM security is a subset focused on large language model applications. It is largely defined by the OWASP Top 10 for LLM Applications, which deals with prompt injection, system prompt leakage, vector and embedding weaknesses, and similar issues that only exist once you put a language model behind an interface.
AI application security is the practical intersection of the two: securing the application that wraps the model, including the web front end, the API layer, the authentication and authorisation around it, and the code that handles its output. It is where most real-world incidents will land, because that is where data moves.
If you are building a product, you need AI security and AI application security. If you are running a chatbot or assistant over documents, LLM security is your day-to-day. All of them sit inside your existing cyber security programme.
Why AI security matters for UK organisations
The cost case is now clear. IBM’s Cost of a Data Breach report puts the global average cost of a breach at US$4.99 million in 2026, a record high and a 12% rise year on year. The same research records a 56% increase in AI-driven attacks, led by deepfake impersonation and AI-enabled malware, and puts the average cost of an AI model inversion attack at US$6 million.
The threat mix has shifted with it. Verizon’s 2026 Data Breach Investigations Report found that 31% of breaches now begin with exploitation of a software vulnerability, overtaking stolen credentials as the leading initial access route, with 48% involving ransomware and 15% of attack techniques showing signs of generative AI augmentation.
For UK organisations specifically, the gap is assurance rather than awareness. The official breach survey shows 43% of businesses reporting an attack in the previous 12 months, 31% with a board member responsible for cyber security, 15% reviewing the cyber risk of immediate suppliers, and just 6% reviewing the wider supply chain.
Those last two figures matter more in an AI context than they do elsewhere. A model is only as trustworthy as the data it was trained on, the model weights you downloaded, the libraries in the pipeline, and the APIs you call at inference time. A supplier risk review that stops at the immediate contract misses most of that chain. Our guide to AI and ML security services covers how we approach that assessment in practice.
The AI security risks that matter most
The risk list is shorter than the marketing suggests. Most incidents trace back to a handful of mechanisms, and the OWASP Top 10 for LLM Applications 2025 is a reasonable map of the application-layer ones.
| model | What it looks like in practice | Primary controls |
|---|---|---|
| Prompt injection | Untrusted content in a document or web page instructs the model to ignore its rules and act on behalf of the attacker | Treat all model input as untrusted, separate instructions from data, constrain what the model can do with its output |
| Sensitive information disclosure | The model reveals training data, system prompts or the content of another tenant through a crafted request | Data minimisation, tenant isolation, output filtering, retrieval access control |
| Supply chain compromise | A poisoned model, dataset or dependency enters the pipeline unnoticed | Model and dataset provenance, AI bill of materials, dependency scanning, signed artefacts |
| Data and model poisoning | Manipulated training or retrieval data shifts behaviour or plants a backdoor | Data validation, source trust, drift and anomaly detection, retraining controls |
| Excessive agency | An agent with broad tool access takes a destructive or unauthorised action from a prompt it should not have trusted | Least privilege per tool, human approval for high-impact actions, action allowlists |
| Unbounded consumption | Crafted input drives runaway inference cost or denial of service | Rate limits, token and cost ceilings, input size limits, monitoring |
| Shadow AI | Staff paste confidential data into unapproved tools, or wire unvetted models into workflows | Acceptable use policy, approved tooling, data loss prevention, training |
The mechanisms behind these are catalogued in detail by MITRE ATLAS, the adversarial threat landscape for AI systems, which in its September 2026 release documents 16 tactics, 208 techniques, 40 mitigations and 73 case studies. ATLAS is the closest thing the field has to a shared vocabulary, and it is worth using for threat modelling rather than inventing your own taxonomy. Our threat modelling assessment applies that kind of structured approach, including AI-specific abuse cases.
Two risks deserve a specific mention for UK organisations. Shadow AI is the fastest-moving one, because it is a policy and culture problem rather than a technical one, the NCSC blog The hidden risks of shadow AI (September 2026) is a good internal briefing. Model inversion, where an attacker reconstructs training data from a model’s outputs, is the one with the clearest financial signature; IBM’s US$6 million average cost makes it the most expensive AI-specific attack in the current data.
How to secure AI systems: a practical control set
You do not need an AI-specific security programme bolted onto the side of your existing one. You need eight things, done deliberately.
- Inventory every model, dataset and embedding store. You cannot secure what you have not recorded. Keep an AI register with the model, its provider, version, purpose, data sources and owner. This is also the prerequisite for an AI bill of materials.
- Threat model the AI system, not just the application. Extend your existing threat modelling to cover prompt injection paths, retrieval sources, tool permissions and the trust boundaries between them. The NCSC’s Guidelines for secure AI system development break this into secure design, secure development, secure deployment, and secure operation and maintenance.
- Apply least privilege to data and tools. A model should be able to reach only the data its purpose requires, and an agent should hold only the permissions that task needs. This single control neutralises a large share of excessive agency and disclosure risk. Retrieval and vector stores need the same access discipline as any other data tier.
- Treat all model input as untrusted and all output as unverified. Content that reaches the model from a web page, email, document or user field is an instruction channel. Validate and sanitise output before it is executed, rendered or passed to another system.
- Constrain agency. Allowlist the actions an agent can take, require human approval for anything irreversible or high value, and set hard ceilings on the number of steps, tokens and spend per request.
- Secure the supply chain. Record where weights, datasets and libraries come from, prefer signed artefacts from trusted registries, scan dependencies, and review the security posture of model and API providers rather than assuming it.
- Log, monitor and detect drift. Log model inputs, outputs, tool calls and refusals; alert on volume anomalies, repeated policy violations and behaviour change over time. Drift and poisoning show up as statistical signals long before they show up as incidents.
- Write an AI incident response plan and test it. Decide in advance how you will disable a model, roll back to a previous version, notify the ICO within 72 hours where personal data is affected, and communicate with customers. Then rehearse it.
None of this is exotic. It is asset management, least privilege, supply chain oversight, logging and incident response, the same disciplines you already run, extended to a new class of asset. That is also why it is achievable without a specialist team, provided somebody owns it.
Standards, regulation and frameworks
The UK has moved from principles to something closer to testable requirements, and 2026 is the year the picture became concrete.
NCSC secure AI guidelines are the foundation. Published in November 2023 and endorsed by 19 international partners, they cover secure design, secure development, secure deployment, and secure operation and maintenance, and take a secure-by-default stance.
The UK Code of Practice for the Cyber Security of AI, published by DSIT and the NCSC on 31 January 2025, turns that into 13 voluntary principles across five lifecycle phases. In January 2026 it was published as a formal standard, ETSI EN 304 223, giving suppliers something concrete to be assessed against.
NIST AI RMF 1.0 structures risk work around four functions, Govern, Map, Measure and Manage, with the Generative AI Profile (NIST AI 600-1) adding generative-specific guidance. It is voluntary but widely referenced, and it maps cleanly onto ISO/IEC 42001, the AI management system standard, if you want a certifiable management system rather than a framework. Our NIST services and regulatory compliance support cover both routes.
The EU AI Act applies from 2 August 2026, following the AI Omnibus that entered into force on 27 July 2026 and moved high-risk obligations for Annex III systems to 2 December 2027 and Annex I products to 2 August 2028. If you sell into the EU, it is a product requirement, not just a compliance one.
The ICO’s AI and data protection guidance, including its AI risk toolkit, remains the reference for the UK GDPR side of the equation, and the UK AI Security Institute continues to publish research on frontier model capability, having evaluated more than 30 systems since November 2023.
Common mistakes
- Treating a model as a black box you cannot test. Model behaviour is observable, loggable and testable. Not testing it is a choice, not a limitation.
- Securing the model and ignoring the application. Most exploitable conditions sit in the API layer, the retrieval access control and the output handling, not in the weights.
- Assuming the provider’s security is your security. Availability of the model does not transfer responsibility for how you use it or what data you send it.
- Leaving agency unconstrained because the use case is internal. Internal agents with broad tool access are a lateral movement path.
- Adopting AI before the acceptable use policy exists. Shadow AI is a governance failure first and a technical one second.
Frequently asked questions
Is AI security different from cyber security?
It shares the same foundations but adds AI-specific risks: prompt injection, data and model poisoning, model inversion, excessive agency, and unbounded consumption. It also adds lifecycle stages that conventional security rarely covers, such as training data governance and model decommissioning.
What is the biggest AI security risk right now?
For LLM applications, prompt injection is the most common and the hardest to eliminate, because it exploits the fact that models cannot fully separate instructions from data. In financial terms, model inversion attacks carry the largest average cost, at US$6 million according to IBM’s 2026 research.
How do we secure AI systems on a limited budget?
Start with inventory and least privilege. Knowing which models you run, what data they can reach, and what tools they can call removes a disproportionate amount of risk for very little spend, before any specialist tooling is needed.
Do UK organisations have to comply with the Code of Practice for the Cyber Security of AI?
The code is voluntary, but it is being adopted through procurement and now exists as the ETSI EN 304 223 standard. Suppliers to regulated sectors and to government should expect it to be asked for, in the same way Cyber Essentials became a tender requirement.
How do we stop staff using unapproved AI tools?
Publish an acceptable use policy that says what is permitted and why, provide an approved alternative so the policy is workable, and use data loss prevention to catch confidential data leaving through browser-based tools. Detection without an approved path tends to fail.
Getting started
AI security is achievable incrementally. The organisations that get it right start with a register of what they run, a threat model of how the AI system could be attacked, and one control that closes the biggest gap, usually least privilege on data and tools.
If you need a starting point, Falx works with UK organisations on AI and ML security, from threat modelling and AI application security reviews through to establishing a governance and assurance programme that maps to NCSC guidance, the UK code of practice and ETSI EN 304 223. You can also read our breakdown of the OWASP-aligned CyberSecEval 2 benchmark for how model risk is measured in practice.