← All articles

Cybersecurity

Threat-modelling an AI pipeline before the demo

AI demos optimise for “look what it can do.” Security work asks a colder question: what can an adversary make it do, and what can they learn while it fails? If you wait until after the launch party to answer that, you inherit a system whose trust assumptions were never written down.

This note is a compact threat-modelling checklist for LLM-centred pipelines — retrieval, tools, agents — aimed at teams that already ship software and need to extend their existing practice, not invent a new religion.

Draw the pipeline, then name the assets

Start with a single diagram: user → gateway → model → retrieval / tools → downstream systems → logs. On that diagram, mark:

  • Secrets — API keys, database credentials, signing material.
  • Sensitive content — customer data in the corpus, PII in prompts, proprietary code in context.
  • Privileged actions — anything a tool can write, delete, pay, or unlock.
  • Trust decisions — where output is treated as fact by a human or by another system.

If a box on the diagram can both read secrets and execute tools, treat that box as high-value until proven otherwise.

Attack classes worth listing explicitly

Prompt and instruction injection

Untrusted text (email, PDF, web page, ticket) enters the context and steers the model. Mitigations: strict separation of system instructions vs. retrieved content, output schemas, and refusing to execute tool calls that were only justified by untrusted text without a second check.

Data exfiltration via the model

The model is asked to summarise or “debug” something that includes secrets from retrieval or prior turns. Mitigations: least-privilege corpora, redaction before embedding, session isolation, and blocking known secret patterns in responses.

Tool abuse

An agent with a shell, inbox, or payment API is social-engineered into a harmful call. Mitigations: allow-lists, human approval for irreversible actions, scoped credentials per tool, and rate limits that survive a confused model.

Poisoned retrieval

An attacker plants documents so retrieval consistently surfaces them. Mitigations: provenance on corpus updates, signing or review gates for high-impact sources, and anomaly detection on sudden rank changes.

Supply chain and serving plane

Compromised weights, plugins, or the inference host itself. Mitigations: pinned artefacts, SBOM for the serving stack, network egress controls, and the same patch discipline you already apply to app servers.

The model is not the perimeter. Every tool it can call is part of your attack surface — treat it like an intern with production credentials.

What to cut first

When time is short, prioritise in this order:

  1. Remove or gate tools that can change state outside the chat.
  2. Stop dumping secrets and raw PII into prompts “for convenience.”
  3. Add authentication and authorisation at the gateway — not only at the chat UI.
  4. Log prompts, tool calls, and retrieval IDs with retention limits and access control.
  5. Write an abuse case list and run it before the customer demo, not after.

Closing

You do not need a 40-page threat model for a weekend prototype. You do need an honest diagram, a short list of assets, and a decision about which tools the model is allowed to touch. Do that before the demo hardens into production.

For a structured review of an AI or automation pipeline, contact Hieg.