Created on 2026-03-07 09:00
Published on 2026-03-16 11:15
There was a time when “data leakage” usually meant a bad S3 bucket policy, a stray debug log, or someone exporting a customer table to a spreadsheet named final_v2_REAL.xlsx. Now we have a shinier problem: large language model systems that can leak secrets, internal documents, user information, or regulated data through prompts, memory, logs, retrieval pipelines, tool outputs, and model responses. The magic trick is that everyone feels productive right up until the moment the chatbot becomes the most cheerful insider threat in the company. OWASP now explicitly treats sensitive information disclosure as a major LLM application risk, and privacy guidance has moved in the same direction, emphasizing privacy-by-design, data minimisation, and systematic risk assessment for generative AI systems.
For SRE and DevOps teams, this matters because LLM systems are not just “models.” They are distributed socio-technical systems. They sit on top of identity, logs, data pipelines, vector stores, connectors, CI/CD, tickets, chat platforms, cloud storage, and whatever heroic YAML structure your platform team assembled at 2 a.m. That means leakage is rarely a single bug. It is usually an architectural chain reaction: a prompt includes too much context, retrieval brings back the wrong document, a tool output is trusted too much, logs retain sensitive content too long, and the model politely stitches the whole mess into a perfect answer. From an operations point of view, that is not an AI problem. That is an old reliability problem wearing very fashionable sneakers.
The public conversation often focuses on “the model leaking training data,” and yes, memorisation is a real concern. But in practice, many of the most likely enterprise failures are much more operational and much less glamorous. Sensitive information can show up in the prompt because users paste secrets, tickets, config files, or customer records into chat. It can appear in logs because observability pipelines capture raw prompts and tool traces by default. It can surface in retrieval because the permissions model for the source system did not survive the trip into embeddings. It can be exposed by tools because agents can read email, tickets, files, or internal systems and then echo too much back. It can also be triggered indirectly through prompt injection, where malicious content hidden in a webpage, file, or message tricks the model into revealing or transmitting data it should not touch. OWASP ranks prompt injection as a top LLM risk, and both Google and OpenAI describe data exfiltration as a central failure mode for agentic systems exposed to untrusted content.
This is where human nature enters the scene, as it always does in IT. Engineers under pressure optimise for flow. Product teams optimise for adoption. Security teams optimise for regret avoidance. Nobody wakes up wanting to build a leaky AI assistant; they just want the demo to work before the steering committee meeting. So somebody grants broader access “temporarily,” someone keeps full prompts for debugging “for now,” someone disables a filter because it was “blocking useful answers,” and suddenly the architecture has the emotional maturity of a toddler holding production credentials. That is not cynicism. That is just change management with better autocomplete.
One camp says sensitive data leakage is a serious but manageable engineering problem. This view is not naive. It is backed by real progress in layered defenses: input and output filtering, prompt injection detection, access scoping, DLP controls, stronger evaluation, red teaming, and tighter tool permissions. Google has described a layered defense strategy for prompt injection and an evaluation framework to measure indirect prompt injection risk. Microsoft documents protections for Copilot, including prompt injection blocking and Purview-based governance controls. OpenAI guidance for agents similarly stresses limiting access to sensitive data, enabling only needed apps, and carefully reviewing consequential actions. The basic argument is familiar to SREs: you do not eliminate all failure; you reduce blast radius, improve detection, and avoid single points of catastrophe.
The opposing camp says that some leakage paths, especially prompt injection, are not just immature engineering problems but consequences of how current LLMs work. The UK NCSC put this sharply in December 2025, arguing that current LLMs do not enforce a true security boundary between instructions and data inside a prompt. Google has also said a single silver-bullet defense is not expected to solve indirect prompt injection entirely, and Anthropic has written that prompt injection is far from a solved problem, particularly as models take more real-world actions. That view lands hard for anyone who has ever heard “we’ll just add another regex” in a design review and felt their soul leave their body.
Honestly, both camps are right. Leakage is manageable enough for many use cases, and still too structurally dangerous for others. That is the uncomfortable truth. Some applications can tolerate residual risk if they are carefully sandboxed, minimally privileged, and designed so that model output cannot directly trigger irreversible actions. Other applications should not be agentic at all, or should not see certain data, because the downside is too high. In plain English: a chatbot that drafts internal meeting notes is one thing; an agent with wide connector access, open internet browsing, and permission to shuffle customer records around is another thing entirely. Treating those as the same category is how governance documents become performance art.
Privacy and data protection bodies are increasingly clear that generative AI systems must be designed around necessity, proportionality, and accountability rather than “collect everything and pray the prompt template behaves.” NIST’s Generative AI Profile frames risks such as privacy, cybersecurity, and information integrity as part of a broader lifecycle governance challenge. European privacy guidance on LLMs emphasises privacy and data protection risk assessment, privacy-by-design, and controls to prevent unnecessary processing of personal data. UK GDPR guidance on DPIAs is also a useful reality check: when new technologies are likely to create high risk for people’s rights and freedoms, assessment is not a nice-to-have. It is table stakes.
For platform and reliability teams, that means “observability” cannot be a sacred cow anymore. You do not get to say, “We log everything because troubleshooting is hard,” if “everything” includes customer records, API keys, payroll details, or support transcripts. LLM systems force a more mature question: what minimum telemetry preserves operability without turning your log pipeline into a compliance time bomb? That is not anti-observability. It is observability growing up, getting a mortgage, and finally reading the data retention policy.
The first practical approach is ruthless data minimisation at every boundary. Do not send the whole document when a paragraph will do. Do not retrieve ten internal files when two are enough. Do not give an agent access to the user’s whole mailbox because it only needs one booking confirmation. The most reliable secret is still the one the model never saw. This sounds boring, which is usually a good sign in security. OpenAI guidance for agents recommends limiting access to only the sensitive data and credentials needed for the task, while privacy guidance in Europe leans heavily on data protection by design and by default. Boring, in this case, is beautiful.
The second approach is identity-aware retrieval and tool access, not “YOLO RAG.” If your retrieval layer strips away source permissions or your agent can call tools with a giant service account, you have built a data leak with extra latency. Retrieval should inherit source entitlements, and tools should operate with narrowly scoped credentials, short-lived tokens, and explicit user confirmation for anything sensitive or consequential. Google, Microsoft, and OpenAI all converge on this principle in different language: narrow the access surface, separate trusted instructions from untrusted content as much as possible, and assume connectors raise the stakes dramatically.
The third approach is to treat prompts, retrieved content, and tool outputs as untrusted inputs that require inspection and containment. This is where prompt injection defenses, output classifiers, DLP, and content inspection belong. Microsoft’s Prompt Shields and Purview controls are examples of this product direction, and Google’s layered defense work makes the same case from a different angle. The key operational lesson is not “find the perfect detector.” It is “put multiple ugly, imperfect controls in series so that one bad model judgment does not become one very expensive incident.” Defense in depth may not be glamorous, but neither is explaining to Legal why the assistant quoted an internal salary spreadsheet.
The fourth approach is to redesign logging and tracing for privacy. Store hashes or references where possible. Mask secrets before persistence. Set short retention on raw prompts and tool traces. Separate developer debugging from production telemetry. Apply secret scanning and DLP to observability stores, not just code repositories. SRE teams already know that logs are both a lifesaver and a liability; LLM systems simply make the trade-off less hypothetical. If your traces are detailed enough to reconstruct a customer’s entire support case, congratulations on your observability maturity and condolences on your upcoming audit.
The fifth approach is adversarial testing as an ongoing reliability discipline, not an annual ritual performed shortly before somebody says “go live.” Microsoft recommends adversarial testing for agents, Google has built automated indirect prompt injection evaluation, and the broader industry has moved toward red teaming as a core control. SREs should recognize the pattern immediately. This is chaos engineering for trust boundaries. You are not testing whether the app is fast. You are testing whether it panics responsibly when a malicious PDF, web page, prompt, or tool response tries to talk it into being stupid.
Sensitive data leakage in LLM systems is not one bug, one control, or one team’s fault. It is the collision point between modern AI capability, old-school access control, observability habits, and very human organisational incentives. The mature response is neither panic nor hype. It is architecture. It is constraints. It is choosing use cases that can survive residual risk, and refusing the ones that cannot. In SRE terms, the goal is not to make the system “smart.” The goal is to make failure non-catastrophic, detectable, and survivable. That may sound less exciting than “autonomous AI transformation,” but so does every good production strategy right before it saves your weekend.
And that, perhaps, is the real lesson: the safest LLM is not the one that promises never to leak. It is the one designed so that, when it gets weird, it cannot take the company secrets on a little adventure.
OWASP Top 10 for Large Language Model Applications — LLM06: Sensitive Information Disclosure — https://owasp.org/www-project-top-10-for-large-language-model-applications/
NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative AI Profile — https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
AI Privacy Risks & Mitigations – Large Language Models (LLMs) — https://www.edpb.europa.eu/system/files/2025-04/ai-privacy-risks-and-mitigations-in-llms.pdf
Prompt injection is not SQL injection (it may be worse) — https://www.ncsc.gov.uk/blog-post/prompt-injection-is-not-sql-injection
Mitigating prompt injection attacks with a layered defense strategy — https://security.googleblog.com/2025/06/mitigating-prompt-injection-attacks.html
#SRE #SiteReliability #DEVOPS #LLM #GenerativeAI #AIEngineering #CyberSecurity #DataProtection #PrivacyEngineering #PromptInjection #RAG #MLOps #PlatformEngineering #SecurityEngineering