Building Internal Agents
The hard part is not choosing a model. It is deciding what authority software should receive. A demo can summarize a document or call an API. A production agent must preserve the requesting employee's identity, select only permitted tools, verify state changes, expose its trace, and stop when uncertainty or risk crosses a defined boundary.
Key takeaways
- Start with one bounded workflow whose outcome can be checked independently.
- Treat identity, authorization, and approval as part of every tool call—not as a wrapper around the chat interface.
- Separate durable company knowledge from per-run state and retrieved evidence.
- Evaluate final outcomes, intermediate tool calls, security behavior, latency, and cost.
- Increase autonomy only after shadow-mode traces prove the agent fails safely.
When should you build an internal agent?
Build an internal agent when a valuable workflow needs judgment over unstructured information, has several possible paths, and produces an outcome that another system or person can verify. Use ordinary code when the path is stable. Use retrieval plus generation when the job only needs an answer. Agentic control is justified when the model must choose and sequence actions.
The first design document should describe the work, not the AI. Write down the trigger, inputs, permitted actions, completion evidence, irreversible effects, and escalation owner. OpenAI's current practical guide to building agents similarly recommends agentic systems for complex decisions, brittle rules, and workflows dominated by unstructured data; otherwise, deterministic automation can be enough.
A useful candidate passes four tests:
- Bounded value: completion saves meaningful human effort or reduces queue time.
- Observable finish: a database state, ticket transition, approved artifact, or test result proves completion.
- Recoverable failure: retries are idempotent, or compensation can reverse the action.
- Owned exceptions: a named team handles ambiguous inputs and failed runs.
Do not begin with “answer anything about the company.” Begin with “triage a support ticket, collect the relevant account evidence, draft a response, and route exceptions.” The narrower contract creates better tools, cleaner eval cases, and a permission model that security teams can inspect.
What architecture does an internal agent need?
An internal agent needs six production boundaries: an authenticated entry point, a versioned instruction layer, scoped retrieval, typed tools, explicit run state, and a trace that records decisions and effects. The language model sits inside those boundaries. It should not own credentials, invent authorization, or treat its conversation history as the system of record.
The minimum production path
- Authenticate the caller. Resolve the employee, tenant, role, and delegated scopes before the model runs.
- Assemble context. Load the task, policy version, relevant records, and a small set of retrieved passages.
- Plan and call tools. Let the model select from typed, narrowly described operations.
- Enforce policy outside the model. Check authorization, arguments, rate limits, and approval requirements in code.
- Verify the result. Query the source system again or run a deterministic validator.
- Record the run. Store model, prompt version, tool inputs and outputs, approvals, latency, cost, and final status.
Anthropic distinguishes predefined workflows from agents that dynamically control tool use, and advises teams to start with simple, composable patterns. That distinction matters. A routed workflow is often easier to test than an open-ended loop. Add model-directed planning only where branching logic has become the actual bottleneck.
Use a tool contract that carries user identity and makes side effects obvious:
{
"name": "create_refund_request",
"input": {
"order_id": "ord_123",
"amount_minor": 4900,
"reason_code": "duplicate_charge"
},
"delegated_actor": "employee_42",
"approval_policy": "finance_review_over_limit",
"idempotency_key": "run_8f3:refund:ord_123"
}The model proposes this call. Application code authorizes it. A human approves it when policy requires. The downstream service executes it once. That separation is the production contract.
The execution wrapper should remain boring code. Resolve the delegated actor from the authenticated session; never accept it from model output. Then authorize the named operation, validate its typed arguments, request approval when the policy engine says so, execute with an idempotency key, and read the affected record back. Mark the run complete only when that second read satisfies the workflow's finish predicate.
proposal = agent.next(run_context)
assert proposal.tool in registry.allowed_for(actor)
args = registry.validate(proposal.tool, proposal.arguments)
policy.require_approval(actor, proposal.tool, args)
result = registry.execute(proposal.tool, args, idempotency_key=step_id)
evidence = registry.verify(proposal.tool, result)
run.record(proposal, result, evidence)This wrapper also creates a clean testing seam. QA can substitute a fake registry, return a timeout after a successful write, and confirm that the retry uses the same idempotency key instead of creating a duplicate side effect.
How should retrieval and memory work?
Retrieval should supply evidence for the current decision; memory should preserve only state that has an explicit owner, retention rule, and correction path. Do not pour Slack, tickets, and documents into one vector index and call it company memory. Preserve source permissions, attach citations, filter by tenant and role, and prefer live systems for volatile facts.
Anthropic's context-engineering guidance treats the context window as a finite resource assembled from instructions, tools, external data, and message history. For internal systems, context quality depends on four separate stores:
Store | Purpose | Example | Main control |
|---|---|---|---|
Policy | Stable operating rules | refund policy version | version and effective date |
Knowledge | Searchable evidence | runbooks, product docs | document ACLs and citations |
Run state | Current execution | selected ticket, completed steps | durable checkpoint |
Learned preference | Approved adaptation | team formatting convention | owner, expiry, deletion path |
A searchable knowledge base can organize maintained procedures and product documentation before those sources enter retrieval. The agent should still return document identifiers and quoted evidence to the application layer; retrieval is not authorization, and semantic similarity is not proof that a user can view a record.
Keep context assembly deterministic where possible. Filter first by identity and metadata, then search. Cap retrieved passages. Reject stale policy versions. For high-impact decisions, require the agent to name the evidence it used and let a validator confirm those sources were present in the authorized retrieval set.
How do you secure tools and approvals?
Secure agent tools as delegated APIs: grant the minimum action and data scope, validate every argument, isolate execution, and require approval for sensitive or irreversible effects. Prompt instructions are behavior guidance, not an access-control system. A compromised document, tool result, or user message can redirect a model, so enforcement must remain outside the context window.
OWASP names excessive agency as a risk created by unnecessary functionality, permissions, or autonomy. The practical response is capability design:
- Replace
execute_sqlwith named read operations and constrained mutations. - Issue short-lived credentials for the current user and run.
- Separate read tools from write tools; default new deployments to read-only.
- Put payment, deletion, access grants, and outbound communication behind approval.
- Use allowlists for destinations, repositories, tables, and message recipients.
- Treat tool output as untrusted input that can contain indirect prompt injection.
- Add iteration, time, token, and spend limits to every run.
The NIST Generative AI Profile frames risk work across design, development, use, and evaluation rather than as a launch checklist. Apply that lifecycle view to every new tool. A tool changes the agent's authority and threat model even when the system prompt stays identical.
For development, expose dependencies through narrow test doubles. Zunoy MockAPI can model expected API responses and failure codes before an agent touches a real internal service. The point is controlled integration testing: timeouts, malformed payloads, duplicate calls, permission denials, and partial success should all appear in the test set.
How do you evaluate an agent before rollout?
Evaluate an internal agent on completed outcomes and failure behavior, not on whether its final message sounds convincing. Build a versioned task set from real workflow cases, replay it on every model, prompt, retrieval, or tool change, and inspect both the final state and the path taken. A passing answer can still hide an unauthorized or wasteful trace.
Track at least these layers:
Layer | Example measure | What failure reveals |
|---|---|---|
Outcome | ticket routed to correct queue | task definition or reasoning defect |
Tool use | allowed tool and valid arguments | description or policy defect |
Grounding | claims supported by retrieved record | retrieval or citation defect |
Safety | prohibited action blocked | authorization or approval defect |
Efficiency | turns, latency, and cost per success | looping or model-selection defect |
Recovery | retry produces one side effect | idempotency or state defect |
Run three evaluation modes. Offline cases catch regressions cheaply. Shadow mode processes live inputs without taking actions. A limited pilot allows low-risk writes for a small user group with immediate rollback and trace review. Promotion between modes should depend on explicit error budgets, not a calendar date.
Agent bugs need reproducible evidence: input, retrieved passages, instruction version, full tool trace, expected outcome, and actual state. A structured bug-reporting workflow helps engineering and QA preserve that context while model behavior and dependencies change.
What should the first production release include?
The first production release should contain one agent, one owned workflow, read-only or low-risk tools, durable run state, deterministic completion checks, and an escalation path. Ship the smallest authority that can prove value. Multi-agent orchestration, broad memory, and self-modifying skills belong after traces show a concrete limitation that simpler code cannot solve.
Before launch, require this release gate:
- The workflow charter names the owner, users, value, forbidden actions, and finish condition.
- Every tool enforces delegated identity, typed input, least privilege, and idempotency.
- Retrieval preserves document permissions and returns traceable evidence.
- High-risk effects require approval; all runs have hard stop limits.
- The evaluation set includes normal, adversarial, stale-data, timeout, and duplicate-call cases.
- Dashboards show success, escalation, policy blocks, latency, and cost per verified outcome.
- On-call staff can replay a run, disable a tool, and roll back a release.
Building internal agents becomes manageable when autonomy is treated as earned authority. Start with discovery, make the system prove what it read and did, then widen permissions one measured step at a time. If your team needs supporting tools for documented knowledge, mocked dependencies, or trace-rich defect capture, explore the relevant capabilities in Zunoy's product suite alongside your agent stack.
