MULTI-AGENT COLLABORATION & GOVERNANCE

Your Multi-Agent System for Order Exceptions

Specialized autonomous agents collaborate across carrier telemetry, regional inventory, customer retention, and finance policies to resolve fulfillment delays within human guardrails.

Detect exception
Investigate across systems
Recommend action
Drive to resolution

Open Exceptions

12 +3

vs yesterday

Avg. Resolution Time

1.8 hrs -42%

vs last week

Auto-Resolved

68% +12%

vs last week

Customer Impacted

6 +2

at risk of SLA breach

Delayed Orders 10 exceptions

Total: 10 0 Auto-resolved 0 Escalated 10 Pending
Select an order to view details and investigate.
HUMAN-IN-THE-LOOP CONTROL

Manual Approval Queue

Review, filter by exception category, and approve cases at once that exceeded autonomous agent boundaries.

Loading...

Agent Execution Traces

Loading...
Select a trace to view details.

Performance Dashboard

Loading...

Resolution Breakdown

LLM vs Rules

Golden Set Evaluations

Agent Limits & Settings

PRODUCT DEEP-DIVE

How This Agent Works

A complete walkthrough of the architecture, the 8-step pipeline, the decision-limit system, LLM integration, and the evaluation methodology. Built to show a reviewer exactly what's happening under the hood.

The Problem

Late shipments are the single largest source of customer support tickets in e-commerce. When an order misses its promise date, a human ops agent must look up the order, check the customer's tier, search for closer stock, decide on an action, calculate costs, execute it, write an email, and log the case. That's 15–30 minutes per exception.

During peak seasons (Diwali, Black Friday), thousands of exceptions pile up. Human agents burn out, resolution times spike, and customers churn.

The opportunity: An AI agent can handle 80% of these cases in under 2 seconds, while escalating the remaining 20% to human reviewers with a pre-filled recommendation.

The 8-Step Agent Pipeline

Every time you click "Run Agent", the system executes this pipeline. Each step is traced with inputs, outputs, tool used, and millisecond timing.

1

Trigger Check

Rule

Detects that an order is delayed from the tracking feed. If status ≠ "Delayed", the pipeline stops.

2

OMS / CRM Lookup

Lookup

Retrieves order details (item, destination, delay days) and customer profile (name, tier, email). Tier determines escalation rules.

3

Inventory Lookup

Lookup

Checks if any warehouse has stock for this item. Sets a closer_stock flag that determines if reship is viable.

5

Cost Calculator

Calculation

Loads the rate card: wait ₹0, upgrade ₹500, reship ₹1,200, refund ₹2,500. The LLM reads these costs — it never calculates them.

4

Multi-Agent Collaboration & Consensus

Multi-Agent

Dispatches to 4 domain specialists (Logistics, Inventory, Customer Retention, Finance). The Supervisor Arbiter synthesizes their findings and recommendations into a consensus resolution.

6

Execute / Escalate / Block

Action

Applies decision limits. Auto-resolves, escalates to a human, or blocks the action entirely based on cost, confidence, tier, and tracking silence.

7

Draft Email

Drafting

Generates a customer email. Standard tier auto-sends; Plus/Business emails are held for human review.

8

Log Case

Rule

Saves the full execution trace (all 8 steps with timing) to the runs database. Powers the Traces tab.

Decision Limits & Guardrails

The core thesis: AI agents should operate within human-defined boundaries. The limits system means a PM can tune autonomy by adjusting four numbers.

LimitDefaultPurpose
Max Action Cost₹1,500Escalate if the action costs more than this
Min Confidence0.75Escalate if the LLM is less confident than this
Max Refund₹2,000Escalate if a refund exceeds this amount
Max Tracking Silence48 hoursEscalate if tracking has gone silent this long

Three Outcomes

Auto-resolved

All limits satisfied. Agent executes the action and sends the email automatically.

Escalated

A limit was exceeded. Case goes to the Queue for human approval with the agent's recommendation pre-filled.

Blocked

A "never-rule" was violated (e.g. cancelling an order, refund >1.5× policy). Action is completely prevented.

Hard-coded "Never" Rules: Never cancel an order. Never issue a refund exceeding 1.5× the max limit. Never promise a date the carrier hasn't confirmed. These cannot be overridden from Settings.

Multi-Agent Collaboration Architecture

Rather than a monolithic prompt, the system deploys a specialized multi-agent ensemble with domain separation of concerns, orchestrated by a Supervisor Arbiter:

🚚 Logistics & Carrier Agent

Monitors carrier telemetry, checkpoint silence, route congestion, and transit delay days to diagnose package loss probability.

📦 Inventory & Fulfillment Agent

Inspects regional warehouse networks, checks stock proximity to the delivery destination, and evaluates reshipment feasibility.

🤝 Customer Retention Agent

Assesses customer lifetime value, tier SLA expectations (Standard, Plus, Business), churn risk, and drafts personalized white-glove communication.

⚖️ Finance & Policy Agent

Calculates cost trade-offs (reship vs upgrade vs refund), evaluates margin impacts against rate cards, and audits budget thresholds.

🛡️ Supervisor Arbiter & Consensus Synthesis: Aggregates specialist votes, balances trade-offs (e.g. balancing customer satisfaction with warehouse transit cost), applies the SOP policy, and determines whether the case auto-resolves or requires human-in-the-loop escalation.

The Standard Operating Procedure (SOP)

The SOP is a plain-text file (sop.md) injected directly into the LLM prompt. It codifies the company's exception policy:

1. If the shipping delay is 1 day → wait

2. If the shipping delay is 4+ days → refund

3. If the delay is 2–3 days:

a. Closer stock exists and reship is cheaper → reship

b. No closer stock → upgrade_shipping

Key design choice: The SOP is data, not code. A product manager can edit sop.md to change the policy without deploying new code. The LLM reads it fresh on every call.

Golden Set Evaluation (10 Cases)

The Evaluations tab runs the agent against 10 pre-built test cases. Each case carries its own isolated data (orders, CRM, inventory, rate card, limits) so results are fully deterministic.

#CaseDelayTierWhat It TestsExpected
1Standard Wait1 dayStandardBasic 1-day wait ruleAuto-resolved
2Standard Refund4 daysStandard4+ day refund ruleAuto-resolved
3Reship Closer2 daysStandardCloser stock + cheaper reshipAuto-resolved
4Upgrade No Stock2 daysStandardNo stock → upgrade shippingAuto-resolved
5Business Tier2 daysBusinessBusiness always escalatesEscalated
6Silent Tracking1 dayStandard50h silent > 48h limitEscalated
7Refund Over Limit4 daysStandard₹2,200 > ₹2,000 maxEscalated
8Refund Blocked4 daysStandard₹3,500 > 1.5× limitBlocked
9Cost Limit2 daysStandard₹1,600 > ₹1,500 maxEscalated
10Reship 3 Days3 daysStandard3-day delay, stock availableAuto-resolved

Human-in-the-Loop Escalation Flow

Agent makes a decision
↓
Step 6 checks limits
↓ Within limits
Auto-resolved ✅
↓ Limit exceeded
Escalated → Queue
↓
Approve
Reject + reason
↓ Never-rule hit
Blocked 🔴

The Dashboard's override rate (rejections ÷ escalations) is a feedback signal. A high rate means the SOP or limits need tuning.

Tech Stack

LayerTechnologyWhy
BackendPython 3.11 + FastAPIAsync framework with native Pydantic validation
FrontendVanilla HTML / CSS / JSZero build step, served directly by FastAPI
LLMGoogle Gemini (google-genai SDK)Structured JSON output, fast inference
DataJSON files on diskNo database needed for a prototype
HostingRailwayOne-click deploy with env variable support