The most dangerous part of an LLM feature is not the model itself. It is the way the whole system is assembled: prompts, memory, retrieval, tool calls, and policy enforcement.
This is why AI security work often looks less like classic offensive testing and more like a structured systems review. A model can behave beautifully in isolation and still leak data or overstep authority when the surrounding application is weak.
Where the failure really sits
The most common problems I see are not deep research failures or exotic jailbreaks. They are often the basics, done badly:
- hidden context being passed into prompts
- tool permissions wider than the UI suggests
- sensitive documents retrieved without clear boundaries
- prompt instructions overwritten by user input
- logging that exposes secrets or raw tool output
In other words, the app is not merely the chatbot. It is the entire decision pipeline.
Testing a real workflow
When I red team an LLM feature, I want to see whether the system can be manipulated into doing something it should not do. That usually means trying a few well-targeted paths:
- prompt injection via user-controlled text
- policy bypass through retrieval content
- tool misuse when a user asks for a higher privilege action
- data leakage through summarisation or export features
- unsafe assumptions in the surrounding workflow
A model can be extremely capable and still fail if the application treats the LLM as a trusted decision-maker instead of a component.
The operational angle
Security review for AI features should include more than tooling. It should ask questions about ownership, monitoring, and incident response.
If a production agent can read files, send emails, or trigger workflows, there has to be a clear answer for:
- who can approve the action
- what is logged
- what is blocked by default
- how the system can be rolled back if it makes a bad call
The most important idea is not that AI is special. It is that AI systems can amplify mistakes at speed and scale.
What to keep in mind
The best controls are boring, explicit, and easy to explain:
- least privilege on tools
- content boundaries and redaction
- human approval for risky actions
- a simple retention model for history and memory
- clear monitoring with meaningful alerts
At some point every AI product becomes an identity and access problem, not just a modelling problem.