Securing AI Systems

Compilation date

30 September 2026

Publisher ID

study-guide-publisher@0.8.14+873a57a

This study guide explains how to protect AI systems across data, models, applications, operations, and organizational decisions. Its main sources are the US National Institute of Standards and Technology (NIST), the UK National Cyber Security Centre (NCSC), and international partner agencies. It also draws on the OWASP application security community and research papers examining specific attacks, defenses, and their limits.

Contents

Front matter

• Orientation and reading paths
  Shared system settings and reading paths connect the security questions across the book.

Part I: System and threat modeling

A system map and a threat model give technical and management readers the same description of the system, its operators, attacker access, and possible harm.

• Chapter 1: Mapping the AI system
  A system map records the people, components, data, permissions, and evidence that a security decision about an AI assistant depends on.
• Chapter 2: Threat modeling and controls
  A threat record connects an attacker’s access to a possible failure, the controls expected to prevent it, and the evidence needed to test them.

Part II: Attacks on data and models

Training data, requests, retrieved content, and stored copies expose different attack paths and require controls at different boundaries.

• Chapter 3: Training poisoning and model tampering
  Altering records used in training can change a model’s learned behavior. Investigating a suspected change requires evidence connecting the affected model version to its training data and training process.
• Chapter 4: Input attacks and response tampering
  Altering a request or its delivered response can produce a misleading result without changing model weights. Investigating that result requires records of what the caller submitted, what the model received and generated, and what the recipient obtained.
• Chapter 5: RAG and search poisoning
  Adding false information or hostile instructions to retrieved material can change an application’s answers without changing model weights. Stored copies can carry that influence into later requests.
• Chapter 6: Data leakage and privacy attacks
  An AI application can expose protected information through the data it sends, the copies it keeps, or the answers its model returns to specially chosen queries. Each route needs its own evidence, because a control on one route does not cover the others.

Part III: Platform and service compromise

A model that passed evaluation can still be exposed through acquired software, shared infrastructure, or the identities and interfaces of the running service.

• Chapter 7: Supply chain compromise
  A model file from the intended supplier can still execute unwanted code during loading or behave harmfully during use.
• Chapter 8: Isolation on shared infrastructure
  Data processed by an AI service can be exposed or altered through the platform on which it runs.
• Chapter 9: Runtime intrusion and resource abuse
  Exposed interfaces, stolen credentials, unrestricted network access, and excessive work can compromise an AI service or make it unavailable to legitimate users.

Part IV: Untrusted instructions and excess authority

Untrusted text can influence model output, while document access, action authority, and execution remain separate security decisions.

• Chapter 10: Indirect prompt injection
  Untrusted source text can influence model output. Protection depends on how the application interprets that output, what data it accepts, and which resulting actions and information transfers the receiving services permit.
• Chapter 11: Retrieval access and disclosure
  Retrieval controls determine which records may reach the employee, model provider, and other recipients, including after permissions or stored copies change.
• Chapter 12: Agent action authorization
  Action authorization keeps model-generated proposals separate from user, service, and policy decisions that permit consequential changes.
• Chapter 13: Limits on agent execution
  Browser sessions and coding workspaces give agents access to files, credentials, and services, so execution limits and checks on each result must match the authorized task.

Part V: Security evaluation and response

Release tests measure behavior under specific conditions, while live monitoring and incident investigation check controls during use and recovery.

• Chapter 14: Evidence for release decisions
  Release evidence ties each security claim to the tested cases, the stated attacker effort, and the measured uncertainty behind each number.
• Chapter 15: Live monitoring and operating limits
  Release evidence describes one tested configuration, while live requests, data, suppliers, and components keep changing around it.
• Chapter 16: AI incidents: containment and recovery
  Incident response distinguishes model error, policy failure, hostile content, account compromise, and infrastructure compromise while preserving evidence and testing recovery.

Part VI: AI in attack and defense

Claims about AI in cybersecurity require separate evidence for attacker assistance, defensive value, and deployment cost.

• Chapter 17: AI-assisted attacks
  Different evidence supports capability, observed AI use, prevalence, and measured improvement, guiding controls for verified attack paths.
• Chapter 18: Evaluating AI for defense
  A defensive AI assessment measures useful security work while keeping human judgment, data access, and action authority visible.
• Chapter 19: Comparing AI deployment options
  A deployment comparison holds task, data access, quality target, and action limits constant, then measures results and full operating cost for each option.

Part VII: Organizational AI governance

An organization needs to know which AI systems it uses, who can make decisions about them, and which risks justify further security work.

• Chapter 20: Shadow AI and unmanaged use
  An organization-wide adoption process connects visible AI uses to acceptable-use rules, account and connector controls, and employee support.
• Chapter 21: Responsibility and legal duties
  A responsibility map assigns decisions and evidence across developers, providers, deployers, data owners, purchasers, and affected people.
• Chapter 22: Prioritizing security investment
  An investment decision connects measures, evidence, responsible owners, review dates, and remaining risk to staff, money, and time.

Part VIII: Security under changing conditions

Controls and evidence can lose validity when the sector, operating conditions, or system capabilities change.

• Chapter 23: Adapting controls across sectors
  A transfer review states which assets, duties, permissions, and consequences change before applying earlier controls to a new sector.
• Chapter 24: Reassessing security after technology changes
  Changes in physical access, participant trust, persistence, delegated authority, or systemic uncertainty reopen earlier security assumptions.
• Chapter 25: Capstone: defending a system decision
  A whole system review connects architecture, data, model and platform controls, authority, assurance, recovery, cost, and accountability in one decision.

Appendices

• Implementation and research resources
  Tools, case collections, and research groups that support the book’s controls and evidence methods.
• Conditions for robust aggregation
  The assumptions behind Krum, coordinate median, and trimmed mean results, for readers who need to check them before relying on these rules.
• Glossary
  Definitions of terms used across the book, grouped alphabetically.
• References
  Cited sources retain their reference numbers. Broader background is listed separately as further reading.