Agent Improvement Review Checklist

The first version of an AI agent is usually cleaner than the tenth week of using it.

Prompts change. Cron jobs move. Files get renamed. A temporary workaround becomes the quiet hinge of the whole process. The agent still runs, so everyone assumes the operating model is still true.

That is how a reliable agent becomes a mysterious agent.

The fix is not another launch checklist. It is a recurring improvement review that compares what the agent does with what its owner believes it does.

The failure pattern: configured once, trusted forever

Teams often treat agent setup as a project with a finish line. They define the prompt, connect tools, add a schedule, ship the workflow, and move on.

But agents live inside moving systems. Documents drift. Permissions drift. Business rules drift. People change the runtime because Monday needed saving.

If nobody reviews the gap, the company is no longer managing an agent. It is trusting a rumor about an agent.

The operator lesson

An agent improvement review should ask three plain questions:

1. What is the runtime doing that the source of truth does not describe?
2. What does the source of truth expect that the runtime is not doing?
3. What approval boundary is still implicit?

Those questions are not cosmetic. They reveal where reliability, accountability, and safety are starting to separate.

When to run an agent improvement review

Run the review:

• weekly for production agents touching customers, revenue, operations, or executive decisions;
• after any incident, surprising output, failed handoff, or manual rescue;
• before expanding permissions, schedules, integrations, or autonomous action;
• after ownership, source-of-truth, or workflow changes.

If the agent affects consequence, review is not bureaucracy. It is maintenance.

This applies to personal AI agents too. Durable context and source-of-truth boundaries only stay useful when someone checks whether the live agent still matches the operating intent.

The agent improvement review checklist

1. Runtime drift

Compare what the agent actually does with its documented operating contract.

Check:

• schedules, triggers, and cron jobs;
• model and tool settings;
• connected systems;
• permissions;
• output destinations;
• escalation paths.

Decision: update the documentation, update the runtime, or pause the agent until they match.

2. Source-of-truth drift

Ask which files, records, policies, examples, or databases the agent relies on.

Check whether each source is current, authoritative, and still in the expected location. Stale context is not a prompt problem. It is a bad map handed to a fast driver.

3. Ownership and routing

Confirm the human owner, backup owner, escalation channel, and decision rights.

Every production agent should have someone who can answer:

• What is this agent allowed to do?
• What must it never do?
• Who approves changes?
• Who handles incidents?
• Who can stop it?

4. Interface documentation

Document the agent's expected inputs and outputs. Explain how people should use it and report problems.

If only the builder knows how the agent works, the interface is not finished.

5. Approval boundaries

Separate draft, recommendation, and action.

An agent may be allowed to draft a response without being allowed to send it. It may summarize a deal without being allowed to change CRM stage. It may propose a fix without being allowed to merge code.

Write the boundary where motion begins.

6. Context and memory hygiene

Review what the agent remembers, where memory lives, and what should be deleted, corrected, or promoted to durable context. Memory should make the agent more useful, not more confidently haunted.

7. Skill and procedure bloat

Remove routines nobody uses, merge duplicate instructions, and retire brittle skills that encode old workarounds. A smaller operating surface is easier to trust.

8. Outcome quality

Look at recent outputs and misses. Track acceptance rate, correction patterns, handoff quality, incidents, and time saved.

Quality is not a vibe. It is a pattern you can inspect.

9. Follow-up action IDs

Every finding needs an owner, due date, and decision: fix, document, monitor, expand, or stop. A review without follow-up is just a meeting wearing a safety vest.

What good looks like

A healthy review produces a short change log:

• runtime matches source of truth;
• owners and approval gates are explicit;
• stale context is removed or corrected;
• risky permissions are justified;
• output quality is visible;
• next changes have owners.

The agent should become less magical after every review. That is the point.

One action this week

Pick one production or near-production agent. Open its runtime settings and its source-of-truth documentation side by side. Ask:

1. What is the runtime doing that the source of truth does not describe?
2. What does the source of truth expect that the runtime is not doing?
3. What approval boundary is still implicit?

If you cannot answer those, the next improvement is not a new model, tool, or automation. The next improvement is reconciliation.

For the broader cadence around this work, read The Weekly AI Operating Review That Keeps Sprawl From Coming Back. If the runbook and live scheduler disagree, reconcile them before changing the agent. The Automation Binding Reconciliation Card shows how to compare durable intent with runtime configuration. If your agents are producing workflow artifacts that may become external action, pair this review with a send gate before expanding automation. If you are building this for yourself or a team, read Key AI interfaces.