Event-Driven Quote Triage for a Freight Brokerage

• Client: regional 3PL, 11 dispatchers
• Stack: Python 3.12 · FastAPI · Celery + Redis · Postgres · Browserbase + Playwright · Airtable · Anthropic API (Claude Sonnet 4.6) · Slack

The problem

Load quote requests landed in a shared quotes@ inbox, around 40 a day. A dispatcher would read the request, log into the carrier portal (no public API for what they needed), check which trucks were available on that lane, cross-reference an internal rate sheet, and reply with three carriers and a price. Six to nine minutes each, every time.

Their first automation attempt was a Python script on a five-minute cron driving a hosted agent platform. It worked about 70% of the time, cost roughly $0.42 per request, and failed silently. They usually found out from a customer, not a log.

Three root causes, and all three shaped the rebuild:

• Polling. A 0–5 minute delay before anything started, and the cron had no idea what the previous run had done.
• Re-planning on every run. The agent asked an LLM to navigate the portal from scratch each time, so any UI jitter became a fresh failure mode, and every run paid full navigation token cost.
• No durable state. Nothing recorded which requests had been handled, so retries either duplicated replies or never happened.

The architecture

Everything runs as a small Python service on a single Fly.io machine, a FastAPI app that receives Gmail push notifications, and a Celery worker pool that does the actual work.

Key engineering decisions

Real-time trigger, not a cron. Gmail's users.watch publishes to a Google Pub/Sub topic, which pushes to a FastAPI endpoint. The handler does three things and nothing else: verify the JWT on the push request, write the message ID to Postgres, and return 204. Pub/Sub redelivers anything not acked within 10 seconds, so all real work is handed to Celery and the endpoint stays under 50 ms.

@app.post("/gmail/push")
async def gmail_push(req: Request, claims=Depends(verify_pubsub_jwt)):
    envelope = await req.json()
    history_id = decode_message(envelope["message"])["historyId"]
    for msg in gmail.list_since(history_id):
        if requests_repo.claim(msg.thread_id):   # INSERT ... ON CONFLICT DO NOTHING
            process_quote.delay(msg.thread_id)
    return Response(status_code=204)

Idempotency lives in the database, not in memory. claim() is an INSERT ... ON CONFLICT DO NOTHING on the Gmail thread ID. If Pub/Sub redelivers, or the safety net and a live webhook race each other, exactly one worker wins the row. This is the piece that eliminated duplicate replies outright.

Cached navigation instead of per-run LLM planning. Playwright drives a Browserbase session with stored credentials and a persistent context. The first time the portal layout is seen, an LLM produces a navigation plan, a sequence of selectors and actions, which is hashed against the page structure and cached in Postgres. Every subsequent run replays the cached plan with plain Playwright calls: no LLM, no tokens, sub-second steps. Only when a replay step fails does the code fall back to re-planning, write the new plan, and post a Slack notice that the portal changed. That's the same idea as Skyvern's Route Memorization, built explicitly so the failure path is visible rather than silent.

Prompt caching on the ranking call. The system prompt, the scoring rubric, and the serialised carrier roster are static across requests and sit in a cached block; only the parsed request and the fresh availability data go in the user turn. Roughly 11k stable input tokens moved to cache reads.

system=[{
    "type": "text",
    "text": RUBRIC + roster_block,
    "cache_control": {"type": "ephemeral"},
}]

Concurrency capped deliberately. The Celery pool runs three workers. The portal rate-limits aggressively and had previously locked the shared account twice, so throughput was traded for account safety.

State machine and the safety net

Every request row moves through an explicit status column. The hourly sweep is the belt-and-braces layer for the rare case where a Pub/Sub push is lost or a worker dies mid-run.

The sweep is a Celery Beat task, about 30 lines: select rows in processing with claimed_at < now() - interval '15 minutes', reset the attempt counter, re-enqueue. After three attempts it moves to failed and posts a Slack alert with the thread link and the last exception, so a dispatcher can handle it manually and an engineer can see exactly where it broke.

Errors are handled in three tiers rather than one generic retry: transient network and portal timeouts retry with exponential backoff inside the task; a failed selector replay triggers re-planning and a "portal layout changed" notice; anything else fails the row loudly.

Results

The cost drop came from two places roughly equally: cached navigation removed the per-run planning tokens, and prompt caching cut input cost on the ranking call.

Delivered

Dockerised FastAPI + Celery service with a one-command deploy; Postgres schema and migrations; the Browserbase automation module with cached-plan replay and fallback; the Claude ranking module with the rubric prompt; Slack alerting; a runbook covering the four common failure modes and how to re-run a request by thread ID; and a 45-minute handoff call testing a live request end to end.