Human-in-the-Loop
When humans are needed and escalation patterns
Human-in-the-Loop
Human-in-the-Loop (HITL) is when AI agents escalate to humans for oversight, approval, or handling of complex cases.
Why Escalate?
Agents escalate to humans when:
| Reason | Example |
|---|---|
| Uncertainty | Confidence below threshold |
| Complexity | Beyond agent capabilities |
| Policy | Requires human approval |
| Risk | High-stakes decision |
| Novel | New situation, no precedent |
| Customer Request | User asks for human |
| Quality Gate | Random sampling for QA |
Escalation Patterns
Confidence-Based Escalation
Agent checks confidence before proceeding:
interface AgentResult {
status: 'completed' | 'escalated'
confidence: number // 0-1
result?: any
reasoning: string
}
async function performActivity(task: Task): Promise<AgentResult> {
const result = await agent.do(task)
// Check confidence threshold
if (result.confidence < 0.8) {
return {
status: 'escalated',
confidence: result.confidence,
reasoning: 'Confidence below threshold',
result // Include partial result for human review
}
}
return {
status: 'completed',
confidence: result.confidence,
result
}
}Use Cases:
- Complex support tickets
- Ambiguous requirements
- Multiple valid interpretations
- Edge cases
Policy-Based Escalation
Certain activities always require human approval:
interface ApprovalPolicy {
activity: string
requiresApproval: boolean
approver: Worker
conditions?: Condition[]
}
const invoicePolicy: ApprovalPolicy = {
activity: 'invoice.approv.ing.as',
requiresApproval: true,
approver: { $type: 'Human', role: 'finance-manager' },
conditions: [
{ field: 'amount', gte: 10000 } // Only large invoices
]
}
// In workflow
{
name: 'verify-invoice',
activity: 'invoice.verif.ing.as',
actor: 'agent.accounting',
output: 'verification'
},
{
name: 'approve-invoice',
activity: 'invoice.approv.ing.as',
actor: 'human.manager', // Always human for approval
input: {
invoice: '$.trigger.invoice',
verification: '$.verification'
}
}Use Cases:
- Financial approvals
- Legal documents
- Hiring decisions
- Security changes
- Contract signing
Complexity-Based Escalation
Agent recognizes limits of capability:
interface ComplexityCheck {
indicators: string[]
threshold: number
escalateTo: Worker
}
const supportComplexity: ComplexityCheck = {
indicators: [
'multiple-systems-involved',
'custom-integration',
'data-corruption',
'security-incident',
'angry-customer'
],
threshold: 2, // If 2+ indicators present
escalateTo: { $type: 'Human', role: 'senior-support' }
}
async function triageTicket(ticket: Ticket): Promise<Result> {
const indicators = analyzeComplexity(ticket)
if (indicators.length >= supportComplexity.threshold) {
return escalate({
reason: 'complex-issue',
indicators,
to: supportComplexity.escalateTo,
ticket
})
}
// Agent handles simple case
return resolveTicket(ticket)
}Use Cases:
- Multi-system issues
- Custom/edge cases
- Data recovery
- Security incidents
- VIP customers
Risk-Based Escalation
High-stakes decisions require human judgment:
interface RiskAssessment {
impact: 'low' | 'medium' | 'high' | 'critical'
probability: number
riskScore: number
}
function assessRisk(action: Action): RiskAssessment {
const impact = determineImpact(action)
const probability = calculateProbability(action)
const riskScore = impactScore(impact) * probability
return { impact, probability, riskScore }
}
// In activity
async function performAction(action: Action): Promise<Result> {
const risk = assessRisk(action)
// High risk: Require human approval
if (risk.riskScore > 0.7 || risk.impact === 'critical') {
return escalate({
reason: 'high-risk',
risk,
action,
recommendation: await agent.recommend(action)
})
}
// Low risk: Proceed
return execute(action)
}Use Cases:
- Database deletions
- Production deployments
- Access grants
- System changes
- Refunds/credits
Quality-Gate Escalation
Random sampling for quality assurance:
interface QualityGate {
sampleRate: number // % to review
reviewer: Worker
criteria: string[]
}
const contentQuality: QualityGate = {
sampleRate: 0.1, // Review 10%
reviewer: { $type: 'Human', role: 'content-editor' },
criteria: [
'Brand voice consistency',
'Factual accuracy',
'Grammar and style',
'SEO optimization'
]
}
async function generateContent(brief: string): Promise<Result> {
const content = await agent.generate(brief)
// Random sampling
if (Math.random() < contentQuality.sampleRate) {
return escalate({
reason: 'quality-review',
type: 'random-sample',
content,
criteria: contentQuality.criteria,
reviewer: contentQuality.reviewer
})
}
return { status: 'completed', content }
}Use Cases:
- Content generation
- Design work
- Code reviews
- Customer communications
- Automated decisions
Customer-Requested Escalation
User explicitly asks for human:
// In chat interface
interface ChatMessage {
message: string
requestHuman?: boolean
}
async function handleChat(message: ChatMessage): Promise<Response> {
// Check for escalation request
if (message.requestHuman || detectsHumanRequest(message.message)) {
return escalate({
reason: 'customer-requested',
chat: message,
context: getChatHistory()
})
}
// Agent responds
return agent.respond(message)
}
function detectsHumanRequest(message: string): boolean {
const patterns = [
/speak to a person/i,
/talk to a human/i,
/real person/i,
/representative/i,
/agent (isn't|is not) helping/i
]
return patterns.some(pattern => pattern.test(message))
}Use Cases:
- Frustrated customers
- Complex questions
- Sensitive issues
- Preference for human
- Trust concerns
Escalation Flow
Standard Escalation Process
Agent Activity
↓
Check Escalation Criteria
↓
┌───────────────┐
│ Should │
│ Escalate? │
└───────────────┘
↓ ↓
No Yes
↓ ↓
Complete Escalate
↓
Queue for Human
↓
Human Reviews
↓
┌───────────┐
│ Approve, │
│ Reject, or│
│ Modify │
└───────────┘
↓
Complete Activity
↓
Feedback to AgentEscalation Implementation
interface Escalation {
$type: 'Escalation'
$id: string
timestamp: Date
// Context
activity: string
task: Task
agent: Agent
// Why
reason: EscalationReason
confidence?: number
risk?: RiskAssessment
indicators?: string[]
// What
partialResult?: any
recommendation?: any
context: any
// Who
escalatedTo: Worker
priority: 'low' | 'medium' | 'high' | 'urgent'
}
async function escalate(params: EscalationParams): Promise<Escalation> {
const escalation: Escalation = {
$type: 'Escalation',
$id: generateId(),
timestamp: new Date(),
activity: params.activity,
task: params.task,
agent: params.agent,
reason: params.reason,
confidence: params.confidence,
risk: params.risk,
indicators: params.indicators,
partialResult: params.partialResult,
recommendation: params.recommendation,
context: params.context,
escalatedTo: params.escalatedTo,
priority: determinePriority(params)
}
// Queue for human review
await escalationQueue.add(escalation)
// Notify human
await notify(escalation.escalatedTo, {
type: 'escalation',
escalation,
urgent: escalation.priority === 'urgent'
})
// Return escalation for tracking
return escalation
}Human Review Interface
Review Queue
Humans access escalations via review queue:
{activity}.review.ing.as/{context}/{role}
ticket.review.ing.as/support/manager
invoice.review.ing.as/finance/controller
design.review.ing.as/creative/director
code.review.ing.as/engineering/seniorReview UI
ReviewQueue
├── FilterPanel
│ ├── By priority
│ ├── By reason
│ ├── By agent
│ └── By date
│
├── EscalationCard (for each item)
│ ├── Header
│ │ ├── Activity
│ │ ├── Priority badge
│ │ ├── Time waiting
│ │ └── Agent who escalated
│ │
│ ├── ReasonPanel
│ │ ├── Why escalated
│ │ ├── Confidence score
│ │ └── Risk assessment
│ │
│ ├── ContextPanel
│ │ ├── Original task
│ │ ├── Relevant history
│ │ └── Related items
│ │
│ ├── AgentWorkPanel
│ │ ├── Partial result (if any)
│ │ ├── Agent's reasoning
│ │ └── Recommendation
│ │
│ └── ActionPanel
│ ├── Approve (accept agent's work)
│ ├── Modify (edit and approve)
│ ├── Reject (do it yourself)
│ └── Re-assign (send to someone else)
│
└── MetricsPanel
├── Queue depth
├── Avg wait time
├── Resolution rate
└── Escalation trendsReview Actions
type ReviewAction =
| ApproveAction // Accept agent's result
| ModifyAction // Edit and approve
| RejectAction // Human handles completely
| ReassignAction // Send to different human
interface ApproveAction {
type: 'approve'
escalation: Escalation
feedback?: string // Optional feedback to agent
}
interface ModifyAction {
type: 'modify'
escalation: Escalation
changes: any // What was changed
reason: string // Why modified
}
interface RejectAction {
type: 'reject'
escalation: Escalation
reason: string // Why rejected
humanResult: any // Human's work
}
interface ReassignAction {
type: 'reassign'
escalation: Escalation
to: Worker // New reviewer
reason: string // Why reassigned
}
// Handle review
async function handleReview(
escalation: Escalation,
action: ReviewAction
): Promise<Result> {
switch (action.type) {
case 'approve':
// Use agent's result
await complete(escalation.task, escalation.partialResult)
await provideFeedback(escalation.agent, {
type: 'positive',
message: action.feedback
})
break
case 'modify':
// Use modified result
await complete(escalation.task, action.changes)
await provideFeedback(escalation.agent, {
type: 'learning',
original: escalation.partialResult,
improved: action.changes,
reason: action.reason
})
break
case 'reject':
// Use human's result
await complete(escalation.task, action.humanResult)
await provideFeedback(escalation.agent, {
type: 'correction',
incorrect: escalation.partialResult,
correct: action.humanResult,
reason: action.reason
})
break
case 'reassign':
// Send to different human
await escalationQueue.reassign(escalation, action.to)
break
}
// Record metrics
await recordReviewMetrics(escalation, action)
}Escalation Strategies
Tiered Escalation
Route to appropriate level based on complexity:
interface EscalationTier {
level: number
role: string
handles: string[]
}
const supportTiers: EscalationTier[] = [
{
level: 1,
role: 'support-agent',
handles: ['simple', 'common', 'documented']
},
{
level: 2,
role: 'senior-support',
handles: ['complex', 'technical', 'multi-system']
},
{
level: 3,
role: 'engineering',
handles: ['bug', 'data-issue', 'security']
}
]
function determineTier(ticket: Ticket): EscalationTier {
const complexity = analyzeComplexity(ticket)
return supportTiers.find(tier =>
tier.handles.some(type => complexity.includes(type))
) || supportTiers[0]
}
// Escalate to appropriate tier
const tier = determineTier(ticket)
await escalate({
task: ticket,
reason: 'complexity',
escalatedTo: { $type: 'Human', role: tier.role }
})Collaborative Escalation
Agent and human work together:
interface CollaborativeEscalation {
mode: 'assist' | 'review' | 'pair'
}
// Assist mode: Agent does work, human spot-checks
const assist: CollaborativeEscalation = {
mode: 'assist'
}
// Agent: "I generated this contract. Does section 3 look correct?"
// Review mode: Agent provides analysis, human decides
const review: CollaborativeEscalation = {
mode: 'review'
}
// Agent: "Here are 3 options with pros/cons. Which should we use?"
// Pair mode: Agent and human work together real-time
const pair: CollaborativeEscalation = {
mode: 'pair'
}
// Agent: "I'm working on this now. Want to guide me?"Batch Escalation
Group similar items for efficient review:
interface BatchEscalation {
items: Escalation[]
groupedBy: 'reason' | 'type' | 'agent' | 'priority'
batchSize: number
}
// Group escalations
const batches = await escalationQueue.groupBy('reason', {
batchSize: 10
})
// Present as batch for review
batches.forEach(batch => {
displayBatchReview({
title: `${batch.reason} (${batch.items.length} items)`,
items: batch.items,
bulkActions: ['approve-all', 'modify-pattern', 'reject-all']
})
})Feedback Loops
Agent Learning
Humans provide feedback that improves agent performance:
interface Feedback {
escalation: Escalation
review: ReviewAction
metrics: {
correctness: number // Was agent right?
efficiency: number // Was escalation needed?
learning: string // What to improve
}
}
async function provideFeedback(
agent: Agent,
feedback: Feedback
): Promise<void> {
// Record for training
await agent.learn({
task: feedback.escalation.task,
agentResult: feedback.escalation.partialResult,
humanResult: feedback.review.type === 'reject'
? feedback.review.humanResult
: feedback.review.changes,
improvement: feedback.metrics.learning
})
// Adjust confidence calibration
if (feedback.metrics.correctness > 0.8 && feedback.escalation.confidence < 0.8) {
// Agent was too cautious
agent.calibrate({ increaseConfidence: true })
} else if (feedback.metrics.correctness < 0.5 && feedback.escalation.confidence > 0.8) {
// Agent was overconfident
agent.calibrate({ decreaseConfidence: true })
}
}Escalation Reduction
Track and reduce unnecessary escalations:
interface EscalationMetrics {
rate: number // % of tasks escalated
necessaryRate: number // % that truly needed human
wasteRate: number // % that could have been automated
byReason: Record<EscalationReason, number>
byAgent: Record<string, number>
trend: 'increasing' | 'decreasing' | 'stable'
}
// Goal: Reduce waste while maintaining quality
async function optimizeEscalations(metrics: EscalationMetrics): Promise<void> {
// If too many unnecessary escalations
if (metrics.wasteRate > 0.3) {
// Increase agent confidence threshold
// Provide more training examples
// Improve agent capabilities
}
// If too few escalations but quality suffering
if (metrics.rate < 0.05 && qualityMetrics.errorRate > 0.1) {
// Decrease confidence threshold
// Add more safety checks
// Expand escalation criteria
}
}Best Practices
Clear Escalation Criteria
Define when to escalate:
const escalationCriteria = {
// Always escalate
always: [
'amount > $10,000',
'legal document',
'security incident',
'customer explicitly requests'
],
// Never escalate
never: [
'routine task',
'well-documented process',
'low risk',
'high confidence'
],
// Conditional
conditional: [
{
condition: 'confidence < 0.8',
action: 'escalate'
},
{
condition: 'complexity > threshold',
action: 'escalate'
},
{
condition: 'risk > acceptable',
action: 'escalate'
}
]
}Fast Response Times
Minimize wait for escalations:
interface ResponseTimeSLA {
priority: 'urgent' | 'high' | 'medium' | 'low'
targetResponseTime: number // minutes
targetResolutionTime: number
}
const slas: ResponseTimeSLA[] = [
{ priority: 'urgent', targetResponseTime: 15, targetResolutionTime: 60 },
{ priority: 'high', targetResponseTime: 60, targetResolutionTime: 240 },
{ priority: 'medium', targetResponseTime: 240, targetResolutionTime: 480 },
{ priority: 'low', targetResponseTime: 480, targetResolutionTime: 1440 }
]
// Alert if SLA at risk
if (escalation.waitTime > sla.targetResponseTime * 0.8) {
await alert({
type: 'sla-risk',
escalation,
timeRemaining: sla.targetResponseTime - escalation.waitTime
})
}Context Preservation
Provide humans with full context:
interface EscalationContext {
// The task
task: Task
activity: string
// What agent did
agentReasoning: string
agentResult?: any
confidence: number
// Why escalating
reason: EscalationReason
details: any
// History
previousAttempts?: Attempt[]
relatedEscalations?: Escalation[]
customerHistory?: any
// Resources
documentation?: Document[]
similarCases?: Case[]
expertContacts?: Contact[]
}
// Give human everything they need
await displayEscalation({
context,
recommendation: agent.recommendation,
quickActions: suggestActions(context)
})Summary
Human-in-the-Loop enables agents to deliver services with human oversight by:
- Knowing When to Escalate - Confidence, policy, complexity, risk, quality
- Efficient Routing - Right escalation to right human at right time
- Clear Context - Humans get full picture to make informed decisions
- Fast Resolution - Minimize wait times with SLAs and prioritization
- Continuous Improvement - Feedback loops reduce unnecessary escalations
The paradigm: Agents handle 80-95% of work autonomously, escalating the 5-20% that truly needs human judgment - combining scale of automation with quality of human oversight.