The problem
Support agents who handle point-of-sale integration cases diagnose problems across half a dozen internal systems: store status, menu sync jobs, integration health, account history and past cases. The knowledge of what to do next lived in a long troubleshooting playbook, and agents are audited against that playbook. The goal was one workspace that gathers the context for a case and a copilot that answers from the same playbook the agent is graded on.
How it is built
In the browser
Support workspaceReact + TypeScript
- Case context
- Copilot
- Other teams' agents, as panels
On the server
Backend-for-frontendNode
One service between the workspace and everything it needs.
Playbook retrieval
Returns the matching sections and where each one came from.
Company AI gateway
Keys, logging and model changes in one place, with a frontier model behind it.
Internal data services
Store status, menu sync jobs, integration health, account history, past cases.
- A browser app and a backend-for-frontend. The workspace is a React and TypeScript app. A small Node service sits between it and everything else: the company's AI gateway, the internal data services, and the playbook.
- Retrieval lives on the server. The playbook never ships to the browser. It sits behind a retrieval endpoint in the backend, which returns the matching sections with their provenance.
- Provenance comes from retrieval, not from the model. Every grounded answer cites the playbook sections that retrieval returned, so the model never gets to invent a citation.
- The model runs through the company gateway. One place for keys, logging and model changes. Before switching models I prove the route with a few matched requests through the real gateway: a control, the new setting, and one that should fail.
- Other teams' agents plug in as panels. The workspace became the place agents already are, so other teams embed their own AI agents in it behind the same access model instead of shipping another tab.
How I measured it
- Measure retrieval on the real corpus, never a fixture. Running it against the actual playbook surfaced four defects a test fixture had hidden, including a rare-term scoring rule that is backwards for a support corpus (the valuable terms are frequent precisely because they matter) and a size cutoff tuned for a large document that pruned everything in a small one.
- Displacement as the regression metric. I tuned the next retrieval change against it: the change had to rescue questions that used to return nothing without pushing the right answer down for the questions that already worked.
- Usage against the pre-pilot baseline. Activity per active day and the number of people with access both grew well past it.
Rolling it out to an outsourced team
The first outsourced support team to work inside the tool could not get in, and every failure looked like the same generic error. I proved access one layer at a time: identity, device and network trust, application permissions, and finally an internal call that was blocked and rendered as a network error. Each layer got its own check so the next cohort would not repeat the hunt.
Training came from one content module that generates the slide deck, the guides and the reference docs, plus a verifier that checks every generated copy. One rule came out of it: a fix is done only when every copy has been fetched back from where people actually read it and checked. Now I coach the team's live cases against one success metric: the share of cases resolved within a set time.
I built and lead that team, the second outsourced support team I have stood up. The first carried a partner API migration.
What broke, and the fix
| What we saw | Cause | Fix |
|---|---|---|
| The copilot vanished after a refactor | Its panel's wiring went out with an export that looked unused | Mounted it again and added a test |
| An embedded agent answered "Request failed" | The MCP SDK's default 60-second request timeout, never set | Set the ceiling explicitly |
| The outsourced team saw a "network error" | An internal call was blocked, and the app rendered the block as a network failure | Traced it layer by layer; the real error now shows |
| An AI code reviewer cited a test file that doesn't exist | Two AI reviewers invented the same fixture | Checked all 21 review threads against the code: 20 were real, 1 was invented |
What I would change at ten times the load
- Move prompt assembly to the server. Today the browser builds the prompt and the backend passes it on. At scale the prompts belong on the server, versioned, with an eval harness: held-out questions scored before any change ships.
- Trace before you store. Tag every request with a trace ID and its metadata at the gateway, which already keeps the requests and responses, before building any new store for them.
- Degrade on purpose. Per-tenant quotas, a retrieval cache, and a ladder for bad days: a grounded answer, then an ungrounded one with a warning, then playbook search alone.
What I would tell the next team
- "The mutation exists" is not "the agent can do it." Before promising what a copilot can resolve, list the writes it can actually reach from the UI.
- Two agreeing documents are one source. When the runbook and the official guide agree, load the live page before trusting either.
- Put the corpus where you can measure it. Server-side retrieval is what made honest measurement and honest citations possible.