1  Mapping the AI system

A system map records the people, components, data, permissions, and evidence that a security decision about an AI assistant depends on.

Suppose an employee uses an internal support assistant through one text box to find the approval required for a supplier contract. Behind that box, the organization depends on identity services, document stores, an application that connects those services to models, and possibly action services, logs, and people who review changes. A security review needs that larger arrangement to judge what can go wrong and what evidence is available.

Assume the organization operates the application and connected business services, while an external provider runs the model service. The employee can search permitted documents and review a proposed business action. The application cannot execute that action until a separate service checks the employee’s permissions. Other operating arrangements change which party can apply controls and inspect records. They are compared below.

The organization must choose whether employees may use this assistant for policy questions and limited business actions. Security reviewers need to consider both the employees receiving answers and the people responsible for the records and processes the application can affect.

Figure 1.1: An employee’s request enters the organization’s AI application through identity. Retrieval filters passages for the current user’s permissions. Internal and external model routes are alternatives, with disclosure checks before an external transfer. Proposed actions require separate authorization. The label ‘Immutable logs and audit trail (every decision)’ describes a desired property, not an established fact of this setup. Coverage and protection against changes to records require verification. Missing records and provider-held evidence can leave decisions or later data use unknown. The lower panels identify records, measurement units, and risks for investigation.

1.1 System scope and boundaries

The visible request does not show which components receive the question, which records the application sends to the model, or which service can change business state. A system map represents components, actors, stores, trust boundaries, and data or action flows. It may be a diagram or a structured representation. A system record keeps that map with the decision, version, assumptions, and evidence used to review the arrangement. The map separates three nested objects so each question goes to the right place.

Table 1.1: Three nested objects used throughout the book.

Object	What it contains	Question it answers
Model	Learned numeric values and a procedure that turn an input into a score, label, or generated text.	What behavior can this learned component produce from its input?
AI application	Software that uses a model’s output as part of a task. In this setup, it authenticates users, assembles context, validates output, and may request an action.	What data and capabilities does the product expose to the model and user?
Operational AI system	The application and model plus people, policies, records, identities, infrastructure, suppliers, monitoring, and response.	Which components and decisions keep the task authorized and recoverable?

Here, an AI system means the complete set of components and parties whose behavior matters to the decision under review. The U.S. National Institute of Standards and Technology (NIST) AI Risk Management Framework¹ treats AI risk across lifecycle stages and among several actors. NCSC secure-development guidance² also covers systems assembled from internal components, hosted models, external APIs, and several providers.

The relevant boundary depends on the decision. A parser unit test may examine one function. A release decision for the assistant includes the application, the model service, connected records, identities, suppliers, action services, monitoring, and the response process.

The release question for the assumed arrangement is whether employees may use the assistant for policy questions and limited proposals. NIST AI RMF MAP 1.1³ calls for context such as intended purpose, users, deployment setting, assumptions, limits, and possible impacts.

The assumed baseline fixes these facts for the traces that follow:

• The organization operates identity, application, retrieval, logging, and business-action services.

• An external provider runs the model when the application requests an answer.

• An employee may read approved policy and assigned customer or supplier records. The employee may not read payroll or another business unit’s restricted records.

• The provider receives only the question, instructions, and other text that the application sends. Direct access to the document store or action service is outside this setup.

• Generated text does not itself change business state. Execution is permitted only after the action service verifies the employee’s identity, current permissions, and confirmation of the exact operation.

• Provider retention, use for model improvement, processing region, and provider logs remain supplier facts that require evidence from the provider.

These facts create a stable starting point. A later chapter may change one operator, data path, attacker capability, or application capability while keeping the remaining facts fixed.

1.2 Training, inference, and evaluation

Changing a model’s training examples can affect later requests, while changing the text of one request may affect only that interaction. Understanding the difference requires separating how a model learns, how it produces an output, and how its behavior is measured.

A predictive model maps an input to one or more estimated outputs. A fraud model may accept transaction features and return a score. A generative model produces a sequence or structured object. A large language model (LLM) generates text as a sequence of tokens, units such as words or pieces of words. Its context is the input sequence available for the current model call.

Training uses examples, an objective, and an update procedure to change model parameters. A parameter is an adjustable numeric value used by the model. In a neural network, a weight is a numeric coefficient used in a computation. Bias values, which add offsets, are another kind of parameter. PyTorch’s layer documentation⁴ distinguishes a layer’s weight and bias. Model releases often use “weights” as shorthand for the saved learned parameters. Using those values also requires compatible model code and configuration. They are separate from the application’s document records.

For a labeled example, the model first predicts an output. The target is the reference output supplied by that example, such as whether a transaction was fraudulent. A loss function assigns a numeric penalty to the prediction relative to that target. A training objective can use this loss alone or combine it with other terms. In this loss-based setup, the update procedure is intended to reduce the chosen objective across training examples by adjusting parameters. In next-token training, the following token in the source text supplies the target. Other training methods can use demonstrations, preferences between outputs, or other feedback. The training result includes parameters and configuration, not a database of ordinary records that the application can query directly.

Training can have several stages. For many LLMs, pretraining is an initial training stage aimed at learning broad patterns from a large data collection. Fine-tuning continues training from an existing model with selected data and an objective. It can change some or all parameters, including a separate set used with a fixed base model (Hugging Face’s training documentation⁵). It can adapt a model to a task or change how it responds. NIST’s adversarial machine learning report⁶ distinguishes training from later application integration.

One published instruction-following approach illustrates the stages. Ouyang et al.⁷ first fine-tuned GPT-3 on demonstrations. They then used human comparisons of outputs to train a reward model, a model that scores responses according to learned preferences. Those scores guided further training of the response model. This is one approach rather than a required pipeline for every chat model. Its results do not guarantee that the model follows every instruction.

Inference applies fixed parameters to an input and produces an output. For the support assistant, the input contains the user request and application instructions. It may also include conversation history or retrieved text. Application memory around the call, such as logs or caches, may still change.

Evaluation measures behavior on selected cases without updating the parameters during that measurement. An evaluation must state the model version, input set, expected result or scoring rule, and conditions. A result on one set does not establish behavior for every possible input.

Retrieval selects external content for the current task and makes it available to the application. Retrieval does not change model parameters. It changes the context for one call and may create retained copies elsewhere in the system. Selecting a relevant passage does not establish who may read it or where it may be sent. Those permission checks follow in the retrieval example.

The prompt is the instructions and content prepared for one model call. Here it means the assembled input, including the user’s question and any selected context. In a structured chat interface, that input is represented as messages. A message has content and a role identifying its source or intended use, such as user, assistant, or system. A system prompt supplies application instructions for the model’s behavior. In the interface illustrated here, those instructions are carried in a message with the system role.

Hugging Face’s chat-template documentation⁸ shows how this implementation converts role and content fields into the token sequence expected by a particular model. Formats and control tokens differ. These fields distinguish instructions and conversation content in the input format. How an interface protects role assignment and how a model follows instruction priority depend on the implementation. A role label does not grant an employee permission to read a document or authorize a business action. Services make those decisions in this setup.

During text generation, the model converts the available context into scores for a possible next token. A decoding rule selects one token, appends it to the context, and repeats until a stopping rule or the call’s length limit applies. Sampling settings can make repeated calls differ even when the initial text is identical. The generated sequence remains model output. Its meaning, permitted use, and consequences depend on the application and the services that act on it.

Generalization is useful behavior on cases that were not used for parameter updates. Memorization is learning particular training examples or details in a way that can allow the model to reproduce or reveal them later. A model can do both. Carlini et al.⁹ demonstrated recovery of individual training examples from GPT-2 under a specified query procedure. Chapter 6 explains why a disclosure claim needs a defined interface and evidence connecting an output to training data.

The following pseudocode follows one request from session verification to a displayed answer. It assumes that the identity and disclosure checks stop processing on rejection. Error handling, timeouts, and logging are omitted so the order of the checks is visible:

Code example: Pseudocode for a small chat path that uses external inference.

session = identity_service.verify(request.session_token)
context = application.build_context(user=session.user, text=request.text)
disclosure_policy.check(context, destination="external-model")
draft = external_model.generate(context)
return application.validate_and_display(draft)

Authentication, context assembly, transfer, validation, display, logging, timeouts, and error handling belong to the application and related services. Those paths can fail even when the model behaves as its operator intended.

Example

Evaluating a routing prediction. Suppose an evaluator uses the application to send "Reset access for account 417" to a model that selects a support category. The evaluator supplies account-support as the reference label for this test case. It judges the output without updating the model during the request.

Assume the model gives that category a higher score than billing-support, and the application selects it. The result is a routing label. The model parameters and account permissions remain unchanged. The application must still check the label and choose whether to route the request or request more information.

Conclusion. The model performed inference with fixed parameters, while the evaluator compared the selected category with a reference label. The test did not train the model or carry out the requested account change.

1.3 Model operators and application capabilities

The system record identifies the operator for each component and the data or actions available to the application. Those two facts describe an AI system more clearly than one deployment label.

Two questions stay separate:

1. Component operator: who runs each part and what evidence each party can supply.

2. Data and actions available to the application: what the product can read, send, propose, or change.

Three model operation patterns distinguish who runs the model: the organization on its own infrastructure, the organization on rented infrastructure, or a provider behind an interface. The table also distinguishes who manages the application, giving five combined operating arrangements.

A shared service may serve several customer organizations or groups. Here a tenant is one such group’s accounts and associated records. Users within that tenant can still have different permissions. Isolation between tenants depends on the service’s checks. A connector integrates the application with another system. Its access depends on the user or service identity it uses, the granted permissions, and the checks applied by the connected service. The application must also enforce the current user’s access and disclosure rules.

Using a provider’s complete application is a further distinction. The organization may configure accounts, permitted uses, tenant settings, and exposed connectors through the controls the supplier offers. The supplier operates the interface, inference, and supplier-side records. The service and contract determine which controls and records the organization can inspect. Compared with operating its own application, the organization depends more on those exposed controls and supplier evidence.

In the managed application platform arrangement, the provider operates the specified orchestration, search, connectors, tools, or model services. The organization operates its identities and connected source systems and configures the policy settings exposed by the platform. The service terms and configuration determine which component makes each access or action decision.

Table 1.2: Model operation and application operation determine which controls and records each party can access.

Operating arrangement	Organization operates	Another party operates	Main review question
Organization model on own infrastructure	Application, model serving, local infrastructure and access policies	Connected external services, when present	Which administrators and services can reach the workload?
Organization model on rented infrastructure	Application, model serving and workload configuration	Rented host infrastructure and provider management services	Which platform access and evidence remain with the provider?
Organization application with external model	Application, identity, retrieval, action policy and local logs	Model inference and provider-side processing	Which fields are sent and which provider records are available?
Provider-operated application	Configuration of accounts, permitted uses, tenant settings and exposed connector settings	Application, model inference and provider-side records	Which controls and records can the customer configure or inspect?
Managed application platform	Connected source systems, organization identities and exposed policy settings	Specified orchestration, search, connectors, tools and model services	Which component makes each access or action decision?

An organization-operated model can still depend on cloud administrators, firmware, package repositories, acquired weights, and external identity systems. A provider-operated model does not automatically receive the organization’s complete document collection. A complete map identifies the actual transfer and responsibility for each component (NCSC secure-development guidance¹⁰). Physical location alone does not establish who has access, who can change configuration, or what evidence the customer receives.

Application capability is a separate question. A tool is a function or service the application makes available for model-assisted work, such as searching documents or requesting a ticket update. A tool call is a structured request identifying an operation and its arguments. A model that supports tool use can generate that request. Application software checks it and invokes a permitted operation. Hugging Face’s tool-use documentation¹¹ demonstrates the handoff and the return of a tool result to the conversation.

Here an agent is an application loop that uses model output and task state to select a tool or another step. The application processes the result and updates its state for a later model call. The loop ends when the task finishes or a stopping rule applies. NIST’s agent discussion¹² describes this iterative pattern. A document-search loop may remain read-only, while still needing access and disclosure checks. An action-enabled loop also needs the business permission checks used below.

The table distinguishes four capabilities that an application can combine:

Table 1.3: Application capabilities state what data and actions the system can reach.

Application capability	What the application can do	New security question
Chat	Send selected text to a model and display the response	What crossed the boundary, and how will the answer be used?
Retrieval	Search a specified collection and add permitted passages to model context	Which passages may this user read and send to this provider?
Action proposal	Produce structured arguments for a possible operation without executing it	Are the operation, object, destination, and scope correct?
Action submission	Submit an operation to a service that checks current identity and policy, then rejects it or changes business state	Which identity authorizes the actual fields, and what result was recorded?

An application can combine any operating arrangement with several capabilities. The examples below state both facts directly:

Table 1.4: Examples combine an operating arrangement with an application capability.

Example	Operating fact	Capability fact	Main change
Supplier-hosted employee chat	Supplier operates the interface and model	Chat only	The organization depends mainly on product settings and supplier evidence.
Internal retrieval application with external inference	Organization operates retrieval and the application. The provider operates inference	Retrieval	Selected private context crosses an operator boundary.
Managed agent platform	Provider operates specified orchestration or connectors	Action submission	A complete map identifies the provider and organization checks before action.
Organization-operated model and application	Organization operates model serving and application code	Action submission	Model-serving evidence increases together with patching and runtime duties.

Adding retrieval changes data exposure. Adding execution changes possible consequences. Moving inference in-house changes operational responsibility and available evidence. Each change leaves the other decisions open. A complete record of the active choice does not prove that every supplier behaves as described.

Example

Comparing model operators. Suppose the same policy question and permitted passage are sent through two arrangements. In one, an external provider runs the model. In the other, the organization runs it on its own infrastructure. To isolate the operating difference, assume both return identical wording.

The first request transfers the question and passage to the provider. The organization can inspect its outgoing request, but evidence about provider storage or administrator access must come from that provider. In the second arrangement, its own operators manage that storage and access. The displayed answer alone reveals neither difference.

Conclusion. An unchanged answer can conceal a changed transfer, a different operator, and different investigation duties.

1.4 Assets, identities, and trust boundaries

A useful system record specifies concrete objects. At minimum, it includes user and service identities, the application, model service, source systems, indexes, prompts, action services, logs, operators, and provider interfaces.

An asset is something whose disclosure, unauthorized change, loss, or unavailability would matter to the decision owner. NCSC secure AI guidance¹³ includes models, data, prompts, software, documentation, logs, and assessments among the assets that may need protection.

Table 1.5: A working asset inventory connects each object to an access decision and possible loss.

Asset	Locations or forms	Access decision	Example loss
Source records	Repositories, connector copies, indexes, caches	Who may ingest, change, retrieve, and delete each copy?	Restricted text enters an unauthorized context.
Model inputs and outputs	Requests, provider payloads, responses, conversation state	Which fields may reach each operator and user?	Sensitive data leaves the permitted route.
Model and application files	Parameters, adapters, prompts, code, packages and container images	Who may admit, update, deploy and roll back each version?	An unreviewed file or configuration changes behavior.
Credentials and policy	User sessions, service tokens, connector scopes, action rules	Which component checks which identity and object?	A service performs an operation outside the user’s authority.
Evidence	Request, retrieval, authorization, provider, deployment, and incident records	Who can read, join, retain, and verify them?	The organization cannot determine what happened.

The same information may appear in several places. A policy document can exist in a source repository, a search index, a model request, a cache, and a log. Different people or services may read each copy. Its retention and deletion rules may also differ.

User identities and service identities have different jobs. A user identity supplies current person and tenant claims. An ingestion service may read a broad source collection to build an index. An application service may call the model. The retrieval service should check the requesting employee’s current permissions before returning indexed content. The ingestion service’s permission to read a collection does not authorize every employee to read that collection. An action service separately checks permission to change business records.

A trust boundary is a point where data or control crosses between operators, users, tenants, privilege levels, or policy domains that cannot simply trust one another. An application programming interface (API) defines how one software component requests work from another. It marks a relevant trust boundary when the caller and receiver have different permissions or operators.

This inventory lets the reader trace one request without treating every arrow as equivalent. Discovery gaps, such as an undocumented connector or an asset missed by the selected discovery methods, leave the map incomplete.

1.5 Requests to external models

The smallest runtime path begins when the employee submits text. The organization’s application authenticates the session and builds a request. It checks what may leave the organization before calling the external model. The application then records and displays the result. No document connector or business-action service is reachable on this path.

Table 1.6: Evidence produced along a chat request that uses external inference.

Step	Component	State or evidence
1	Identity service	Authenticated user and tenant claims linked to a request ID
2	Application	Selected user text, application instructions, destination, and model version
3	Disclosure check	Allowed and rejected fields before transfer
4	External model service	Provider request identifier and generated output when available
5	Application	Validation result, displayed response, timing, and local request record

If the output says that an action occurred, the statement is still generated text. An authoritative business record would be needed to establish a state change. The organization can inspect its serialized request and local logs. Evidence of receipt and later processing must come from the provider.

Example

Following an external request. Suppose an employee requests a summary of a meeting note that the organization permits this provider to process. The application combines the note with its instructions, checks the destination and outgoing fields, and records the request identifier. The provider returns a summary and a response identifier. The application links them and displays the summary.

These records connect the outgoing text to the displayed answer. They do not show which provider replicas stored the request or when every copy was deleted. Those questions need evidence from the provider.

Conclusion. The local record supports a claim about what the application sent and displayed. It cannot establish every later use of that data.

This path establishes the external model boundary. The next section adds retrieval while preserving the same inference operator.

1.6 Retrieval and permission checks

The application retrieves source passages while processing a request. A relevance score cannot answer the access and authority questions that follow.

The retrieval boundary is the point where stored source content enters the application’s request context. Source writes, user access, provider transfer and instruction authority involve separate checks around that crossing. Four questions locate those checks:

1. Who may add or change source content?

2. Which passages may the current user read?

3. Which passages may the organization send to this model provider?

4. Can retrieved text authorize an action or change application policy?

An embedding is a numeric representation produced by a model. For search, passage and query encoders turn text into comparable vectors, or lists of numbers. Comparing these vectors can find related wording without requiring the same words. With the encoders fixed, building these representations is inference, not further training.

Lewis et al.¹⁴ introduced retrieval-augmented generation (RAG) models that combined a learned generator with a dense external index and retriever. Current products use several architectures under the broad RAG label. This book’s application pattern preserves source identity and access metadata. It checks current user permissions before selecting text. The application sends only permitted context to the model. It does not claim to reproduce the paper’s jointly trained design.

For a compact trace, suppose the employee asks “What is the meal expense cap?” The ingestion service has encoded policy passage P17 and stored its vector with a link to source version 3. A query encoder produces a comparable vector for the question. Similarity search ranks stored vectors and returns P17 as a candidate. Its identifier leads back to the original policy text, source version, and access metadata. Assume current user permission and provider disclosure checks both pass. The application adds the permitted text to model context. The score selected a candidate, not a permission. This design adds stored vectors and links, encoding work, and index-update work. Approximate relevance can still return an unsuitable passage.

Search can return several candidate passages. The table shows possible treatments under different access and disclosure conditions; it does not fix the result of the four-to-two example below:

Table 1.7: Relevance, user authorization, provider disclosure, and instruction authority are separate decisions.

Candidate passage	Relevant	User may read	May reach provider	Treatment
Approved supplier policy	Yes	Yes	Yes	Admit with source identity and version
Assigned customer record	Yes	Yes	Depends on data rule	Apply disclosure policy before transfer
Payroll record	Possibly	No	No	Reject before context assembly
External note containing instructions	Possibly	Possibly	Depends on data rule	Admit only if user access and provider disclosure checks pass; preserve the source and treat its instructions as untrusted content.

Relevance answers whether text may help answer the query. Authorization answers whether this user may read it. A disclosure rule answers whether the organization may send it to the provider. Retrieved text remains data even when it contains sentences that look like instructions.

If a forbidden passage enters the serialized model request, a later refusal to repeat it does not undo the transfer. If provider receipt cannot be observed, the record must say that it is unknown. Chapter 6 follows the resulting transfers and retained copies. Chapter 5 examines hostile source content, and Chapter 11 develops retrieval permission checks.

Example

Selecting permitted context. Suppose an employee requests details on a travel reimbursement limit. Search finds four passages. One belongs to a department the employee cannot access. Another may be read by the employee but may not be sent to the external provider. The remaining two pass both checks.

The application removes the first passage under the user access rule and the second under the provider disclosure rule. It sends the two permitted passages to the model. Assume the returned answer cites those passages. The reduction from four candidates to two follows from access and transfer decisions, not from their search ranking.

Conclusion. Relevance, user permission, and permission to transfer data answer different questions. A useful search result can still be excluded from the model request.

1.7 Generated actions and authorization

To request a purchase, an application needs an operation and specific fields, not just a sentence saying that a purchase is appropriate. The illustrative JSON below describes a request for USD 1,200 of equipment from supplier-204. It is model output awaiting validation and authorization:

Code example: A model-generated proposal before authorization.

{
  "operation": "create_purchase_request",
  "supplier_id": "supplier-204",
  "amount": 1200,
  "currency": "USD",
  "reason": "replacement equipment"
}

Before displaying the proposal, the application validates the operation name and schema, although structural validation cannot establish that the operation is correct or authorized.

In this arrangement, execution requires a separate path. The application displays the exact operation, destination, amount or scope, and source information. The employee confirms those fields. The action service checks the authenticated identity, current permissions, business limits, and retry state. It records either a denial or one state change.

The model supplies candidate arguments. Authority comes from identities, current policy, and the service that records the business state. A sentence in a retrieved document cannot change those rules. Chapter 12 develops this separation for tools and agents.

An operation key identifies one intended business action. The application carries it through the proposal, confirmation, submission, and result. When a service enforces matching retries and prevents duplicate effects under a specified contract, the operation key serves as an idempotency key. An identifier that only links records does not provide that protection. Suppose the action service reserves the key for one employee and one operation while execution is pending. It then retains the completed result for 24 hours after completion. It rejects reuse with different fields. Matching retries and concurrent submissions refer to that same operation and cause at most one business effect while the key is protected. A matching retry after completion returns the recorded result.

The key links records. The service must implement the coordination that protects pending execution and the completed result. Current identity, permission, and exact approval checks still apply before a new effect. After the retention period, duplicate protection is no longer assumed, so an uncertain result must be reconciled before another submission. Chapter 12 illustrates reconciliation after a lost reply and a retry within the retention period.

Example

Checking an attempted payment. Suppose the model produces a reimbursement proposal, and the application shows the employee the amount and destination. The employee confirms those exact fields. Before submission, a retry defect replaces the destination with a different account.

The action service receives the changed request and compares it with the confirmed fields. The assumed policy requires rejection when submitted fields differ from the confirmed fields. The decision and payment records show whether the service actually rejected the request and left the payment state unchanged.

Conclusion. The application submitted an altered request whose destination no longer matched the model proposal or employee confirmation. The receiving service must compare the actual submitted fields with the authorized operation before changing the payment state.

1.8 Evidence across the lifecycle

One request trace shows runtime behavior. A security decision also depends on how the system was designed, acquired, evaluated, deployed, changed, operated, and retired. The UK National Cyber Security Centre (NCSC) guidelines¹⁵ organize secure AI development into secure design, development, deployment, and operation and maintenance. The following table expands that lifecycle into the working sequence used for the organization record.

Table 1.8: Lifecycle questions and evidence for the organization’s AI system.

Stage	Main question	Useful evidence
Plan and design	Is AI suitable for the task, users, data, possible losses, and required approvals?	Approved use case, system map, data rules, threat model, acceptance criteria
Develop and acquire	Which code, models, prompts, packages, and sources enter the system?	Repositories, artifact records, reviews, supplier information
Evaluate	Does the specified path satisfy its claims under relevant conditions?	Inputs, expected decisions, versions, observations, results, known gaps
Deploy	Which versions, identities, routes, and settings are active?	Release record, configuration hash, scopes, rollback target
Operate and respond	What happened, what changed, and can the organization contain and recover?	Joined application, provider, platform, action, and incident records
Update	Did a model, prompt, connector, policy, or provider change invalidate earlier evidence?	Change record, affected tests, new evaluation, approval
Retire	Were routes, credentials, indexes, retained data, and supplier duties closed?	Revocation, deletion, retained evidence, contract closure

Evidence varies with the operating arrangement. An organization can inspect its own context assembly and authorization code. For external inference, it may receive provider documentation and request identifiers. Usage records and contractual commitments provide other evidence. Operating the model exposes more technical state and also creates more patching, monitoring, and recovery duties.

1.9 Exercise: moving search services

Suppose an organization plans to move document indexing and search from its own services to a managed platform. The application, identity service, source repository, external model provider, and business-action service remain unchanged.

Which new data transfers and retained copies belong on the system map? Identify which operator supplies evidence for each one, then compare the result with the worked analysis below.

The migration record identifies which steps move: document parsing, passage and query embedding, index storage, and candidate search. Then it adds the new transfers. Source text and metadata may cross to the managed search provider during ingestion. Query text may cross during search. Search results return to the application before user authorization and model-provider disclosure.

The permission mechanism remains in force. The managed index may store tenant and document metadata, but the organization must still identify where current user authorization runs and how permission changes invalidate cached results.

The migration record also states the evidence expected from each operator: source version, ingestion identity, indexed object identifiers, search request, candidate identifiers, authorization decision, serialized model request, and deletion result. Missing provider evidence remains an explicit limit.

Example

Comparing the search migration. Before the proposed move, document parsing, the search index, and search records are held by the organization. The model provider receives selected passages. After the move, the managed search provider also receives source text and metadata to build the index, plus query text when the application searches it.

The revised map therefore adds a provider, transfers to that provider, retained index and query records, and evidence needed for deletion. The application still checks which search results the employee may read and which passages may reach the model provider. Its permission to execute business actions has not changed.

Conclusion. The new threat analysis must consider the added data paths and provider access. It should not assume that moving search also grants execution authority.

1.10 Connecting system and threat models

The completed system record contains:

• The decision and system version.

• Users and service identities.

• Components and their operators.

• Training, evaluation, and inference roles.

• Stored data and runtime transfers.

• User, disclosure, and action checks.

• Possible state changes.

• Logs and evidence gaps.

• Assumptions that trigger a new review when they change.

Chapter 2 adds a bounded attacker and an unwanted outcome to this record. It then identifies an enforcing component, a test, and the remaining risk.

1.11 Chapter checkpoint

Suppose the organization replaces external inference with its own model service but leaves search and payment services unchanged. Which parts of the system map and evidence record change?

Answer. Model operation, model-request destinations, administrator access, and the records available for investigation change. Search permissions and payment authority do not change merely because the model moves. They still need their own checks.

1.12 Chapter references

1. Elham Tabassi, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (2023), Figures 2-3, pp. 10-11. See MAP 1.1, p. 26. See Appendix A, official publication.

2. UK National Cyber Security Centre, CISA, and international partners, Guidelines for Secure AI System Development, version 1.0 (2023), PDF pp. 5-16, official guidance.

3. Patrick Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” Advances in Neural Information Processing Systems 33 (2020), pp. 9459-9474, conference record.

4. Nicholas Carlini et al., “Extracting Training Data from Large Language Models,” Thirtieth USENIX Security Symposium (2021), pp. 2633-2650, official publication.

5. PyTorch contributors, PyTorch documentation 2.14, torch.nn.Linear, “Variables,” versioned documentation. This layer lists learnable weight and bias separately.

6. Hugging Face contributors, Transformers documentation 5.17.0, “Parameter-efficient fine-tuning,” introduction and “Training,” versioned documentation. This example updates adapter parameters while keeping the base model frozen.

7. Apostol Vassilev et al., Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025 (March 2025), section 3.1.1, report pp. 36-38 for training and application integration, and “Inference-time attacks,” item 4, report p. 39 for the agent cycle, report PDF. The InstructGPT sequence is an example rather than a universal pipeline. Capabilities and permissions depend on the application.

8. Long Ouyang et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems 35 (2022), Figure 2 and section 3.1, conference paper. The procedure concerns InstructGPT and the study’s data and evaluations.

9. Hugging Face contributors, Transformers documentation 5.17.0, “Chat templates,” introduction and “Using apply_chat_template,” versioned documentation. The described formatting is implementation-specific.

10. Hugging Face contributors, Transformers documentation 5.17.0, “Tool use,” “Passing tools” and “Tool-calling Example,” versioned documentation. Caller software handles the generated request and appends the result.

---

1. Elham Tabassi, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (2023), Figures 2 and 3, report pp. 10-11, and Appendix A, source.

2. UK National Cyber Security Centre, CISA, and international partners, Guidelines for Secure AI System Development, version 1.0 (2023), executive summary and introduction, PDF pp. 5-7, source. This is development guidance, not a test of a supplier.

3. Elham Tabassi, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (2023), MAP 1.1, Table 2, report p. 26, source.

4. PyTorch contributors, PyTorch documentation 2.14, torch.nn.Linear, “Variables,” versioned documentation. This layer lists learnable weight and bias separately.

5. Hugging Face contributors, Transformers documentation 5.17.0, “Parameter-efficient fine-tuning,” introduction and “Training,” versioned documentation. This example updates adapter parameters while keeping the base model frozen.

6. Apostol Vassilev et al., Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025 (March 2025), section 3.1.1, report pp. 36-38, official report. The report discusses training and application integration. Its InstructGPT sequence is an example rather than a universal pipeline.

7. Long Ouyang et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems 35 (2022), Figure 2 and section 3.1, conference paper. The procedure concerns InstructGPT and the study’s data and evaluations.

8. Hugging Face contributors, Transformers documentation 5.17.0, “Chat templates,” introduction and “Using apply_chat_template,” versioned documentation. The described formatting is implementation-specific.

9. Nicholas Carlini et al., “Extracting Training Data from Large Language Models,” Thirtieth USENIX Security Symposium, USENIX Association (2021), pp. 2633-2650, abstract and sections 4-6, official publication. The authors recovered and verified individual training examples from GPT-2. The result demonstrates possible disclosure under their procedure, not a universal extraction rate.

10. UK National Cyber Security Centre, CISA, and international partners, Guidelines for Secure AI System Development, version 1.0 (2023), executive summary and introduction, PDF pp. 5-7, source. This is development guidance, not a test of a supplier.

11. Hugging Face contributors, Transformers documentation 5.17.0, “Tool use,” “Passing tools” and “Tool-calling Example,” versioned documentation. Caller software handles the generated request and appends the result.

12. Apostol Vassilev et al., Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025 (March 2025), section 3.1.1, “Inference-time attacks,” item 4, report p. 39, official report. Capabilities and permissions depend on the application.

13. UK National Cyber Security Centre, CISA, and international partners, Guidelines for Secure AI System Development, version 1.0 (2023), “Identify, track and protect your assets,” PDF p. 12, source.

14. Patrick Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” Advances in Neural Information Processing Systems 33 (2020), pp. 9459-9474, abstract, Figure 1, Section 1, and Sections 2.2-2.3 (PDF p. 3), source. The paper combines a pretrained generator with a dense Wikipedia index and retriever. It does not prescribe the authorization design used here.

15. UK National Cyber Security Centre, CISA, and international partners, Guidelines for Secure AI System Development, version 1.0 (2023), executive summary and four guideline areas, PDF pp. 5 and 8-16, source.