Control the Tools, Not the Prompt

An AI agent with access to every internal API is not an assistant. It is an unauthenticated user with a good vocabulary.

Most AI agent demos focus on the prompt. In production, the hard part is the tool boundary.

Tool calling is an authorization problem

The agent needs to know which tools it can call, which parameters are allowed, and whose permissions it is acting under. That is not a prompting concern. It is the same problem as exposing an API to a third party, with one difference: the caller is nondeterministic.

A nondeterministic caller means you cannot test the exact input. You have to design the boundary so that any input the model produces is still safe, bounded, and auditable.

A practical checklist

• Give the agent the smallest set of tools it needs.
• Validate every argument before it reaches a business system.
• Log every tool call with the user, the input, and the result.
• Require human confirmation for irreversible actions: payments, deletions, external emails, production changes.
• Set timeouts, retries, and a fallback path when the model or a tool fails.
• Treat the model output as untrusted input.

Why the boundary matters more than the model

Models change. Prices change. A prompt that works on one version can behave differently on the next. The tool boundary is the part you control, and it is the part that decides whether a mistaken plan becomes a mistaken invoice.

The practical pattern is the same one used for any untrusted client: a narrow interface, explicit permissions, validation on the server, an audit trail, and a human in the loop for anything that cannot be undone.

Semantic Kernel helps, but it is not a security control

Semantic Kernel makes orchestration easier. It gives you plugins, functions, and a consistent way to call a model from .NET. It does not make the permissions safe by default. The plugin boundary is where the design work happens, and it is still your code, running with your service account, against your data.

If a plugin wraps an internal API without its own authorization check, the agent inherits everything that API allows. That is usually far more than the feature needs.

Start with one irreversible action to protect

Pick the action that would be most expensive if the agent got it wrong. Put a human confirmation in front of it. Log it. Then work outward.

Which action should your agent never be allowed to take without a human?