Human-in-the-Loop Patterns for AI Agents

Human-in-the-loop is often pitched as “we’ll review the output before it goes out.” That phrasing tells you the team hasn’t designed it. Done well, HITL is a structured interaction with explicit gates, queues, and feedback loops. Done badly, it’s a bottleneck humans grow to ignore.

Four patterns we deploy.

Pattern 1: approve-before-action gate#

The agent prepares the action; a human approves before execution. Used when blast radius is high — outbound communications, payments, account changes.

Implementation that works:

• Agent produces a structured proposal: the action, the inputs, a one-paragraph rationale, and a list of any unusual signals
• Proposal lands in a queue with SLA (often under 2 hours)
• Reviewer approves, edits, or rejects
• Edits flow back as training signal — not as fine-tuning, but as examples in future prompts

Implementation that fails: a Slack channel where the agent dumps proposals and nobody owns them.

Pattern 2: edit-then-send#

The agent produces a draft; the human edits and sends. Common in sales outreach, support responses, document generation.

The edit itself is the most valuable signal in the system. Capture:

• What did the human change?
• Did they keep the structure or rewrite?
• Was the change a correction or a stylistic preference?

These patterns inform prompt improvements. Don’t waste them.

Pattern 3: escalation on uncertainty#

The agent attempts the task. If its confidence is below threshold, it escalates to human. Threshold is calibrated against an eval set, not chosen arbitrarily.

Confidence signals we use:

• LLM-generated self-assessment (noisy but useful in aggregate)
• Disagreement between two model passes
• Distance from nearest known good case in episodic memory
• Hard rules (e.g., dollar amount above threshold always escalates regardless of confidence)

Most escalations should be < 10% of volume. If you’re at 40%, your eval coverage is too narrow or your task class is too varied.

Pattern 4: sample-and-audit#

The agent acts autonomously. A random sample (5–10%) is queued for human audit after the fact. Used for high-volume, low-blast-radius tasks where 100% review is uneconomical.

Audit findings feed two systems:

• A monthly quality report (drift detection)
• Eval set updates (audit cases that surfaced new failure modes become golden tasks)

Sample-and-audit is the maturity tier most enterprises should aim for after 6–12 months of HITL operation.

The queue problem#

The single most common HITL failure is queue rot: proposals pile up, reviewers stop caring, agents effectively act autonomously without anyone owning the approvals.

Defenses:

• SLA on every queue item. Stale items escalate or auto-reject.
• Reviewer rotation so no single person becomes the bottleneck.
• Clear “what does approve mean” rubric. Reviewers should know what they’re checking for.
• Volume guard. If queue depth exceeds a threshold, slow the agent.

When to remove the human#

After 6–12 months of HITL, you have data. For task classes where:

• Human-edit rate is below 5% (humans rarely change anything)
• Audit defect rate is below 0.5%
• Blast radius is bounded (errors are recoverable)

…remove the gate. Move to sample-and-audit. Don’t remove the gate based on confidence alone — base it on observed human behavior.

What we ship by default#

For agent engagements via our AI & LLM integration service:

• Every high-stakes action starts with approve-before-action
• Edit-then-send for any user-facing content
• Escalation thresholds calibrated to eval results
• Sample-and-audit only for matured task classes
• Queues with SLAs, reviewer rotation, and depth alerts

Human-in-the-loop is part of the architecture. Treat it like one.

---

The question isn’t “will a human review this?” — it’s “what is the human’s job in this loop?” Our team designs HITL workflows that scale. Get in touch.