FGS Global · Strategic communications · Agent evaluation and monitoring
An enterprise AI agent network, released with evidence instead of hope.
Every release of Fergus, FGS Global's internal AI agent network, changed prompts, tools or models, and no one could say whether a change made it better or worse, what it cost, or what it was worth. Consultants use Fergus across the firm's practice areas, from M&A and IPO communications to research and stakeholder analysis. We built the evaluation, release and analytics layer behind it and set the lifecycle every change now follows, from stakeholder alignment to firmwide release and production monitoring. Each change is tested against the packs that cover the capabilities it touches, from a suite of 537 scenarios, and every beta or staging release runs the baseline set. Comparison runs caught a change that cut cost per prompt by 40% with quality unchanged, so FGS can choose models on cost and speed. Quality rose from 76% to 94% on a benchmark built from the issues users raised. In conversations with a usable result, Fergus saves consultants about 2 hours, valued at about €1,000 of consultant time.
Snapshot
- Client
- FGS Global, strategic communications and advisory
- Platform
- Fergus, one agent network with about 30 specialized capabilities (skills and subagents)
- Engagement
- August 2025 to September 2026.
- Scope
- Agent development lifecycle, evaluation framework, release gating, production monitoring design, Agents360 design, productivity measurement, anonymization, quality advisory for teams beyond Fergus, handover
- Stack
- Mastra, TypeScript, LLM judges via LiteLLM, GitHub Actions, GCP, Postgres, Next.js
The challenge
Fergus was already in daily use by consultants. The development team was adding skills and subagents quickly. Each release changed prompts, tools, routing or models.
There was no structured way to answer three questions:
Offline evals
Is this change better or worse than what is in production?
Analytics
How do consultants actually use Fergus, and what does it cost?
Business value
What is it worth to the business?
Why it matters
LLM output varies from run to run. A change can look fine in a demo and still break memory, grounding or routing for a different type of request. In a firm that advises on deals and crises, a confident wrong answer is a real risk.
A reliable agent development lifecycle
We set the structure every agent change follows. Each stage has a clear owner and a clear output, so quality is built in rather than checked at the end.
The lifecycle as a loop
- 1
Stakeholder alignment
- 2
Development and evals
- 3
QA
- 4
Beta testing
- 5
Firmwide release
- 6
Monitoring and online evaluation
Production findings become new test cases for the next cycle
- 01
Stakeholder alignment
Domain experts, developers and QA agree on what the capability must do, the critical scenarios and what counts as unacceptable. The result is a short quality contract.
- 02
Development and evals
Developers build the change and write the evals for it. Each change is versioned and tested locally and on the pull request against a reliability-aware gate.
- 03
QA
QA reviews a curated release pack, run three times per candidate, with full evidence: response, scores, trace and cost.
- 04
Beta testing
A small group of consultants uses the change in real work before anyone else does.
- 05
Firmwide release
The change ships to the whole firm once the evidence and beta feedback hold up.
- 06
Monitoring and online evaluation
We designed and built online evaluation for production conversations. Alerts flag problems, and strong conversations become new test cases for the next cycle.
Our approach
We started narrow and expanded once each layer proved useful.
Foundations
- 01
Map the usage
We began with how consultants used Fergus: drafting, research, analysis, preparation. Real usage set the priorities for what to test first. Access to production data was designed around anonymization, least-privilege extraction and audit logging from day one.
- 02
Start from what costs user trust
We asked consultants where Fergus let them down: wrong facts, lost context, answers that missed the point, the wrong skill for the job. These issues became the core of the benchmark. Every eval traces back to a problem users raised, so the quality score measures what users care about.
Offline evals
- 03
Build a trust and stability baseline
The first test pack covered 11 categories, including factual knowledge, memory, safety and refusals, clarification, context grounding and consulting-specific skills. Scorers checked faithfulness, relevance, hallucination, toxicity and tool routing.
- 04
Calibrate the judges
Before calibration, only 73.8% of test cases passed the automated judges. Most failures were in the judges, not the agent. We rewrote the judge guidelines and adjusted thresholds. The pass rate rose to about 95%, and the remaining failures pointed to real issues. This rate measures the judges. It is not the quality of Fergus and not the quality figure below.
- 05
Scale coverage with the network
As Fergus grew into a multi-agent network, coverage grew with it: 537 scenarios across 25 packs and 20 scorers, including checks that the right skill or subagent was used. Packs are selected by the capabilities a change touches, and every beta or staging release runs the baseline set. The highest-priority cases form a fast set for pull requests.
- 06
Test memory, separately and together
We tested everything Fergus remembers: the current conversation, notes it builds from past conversations, uploaded files, long-term recall and memory across chats. Each was tested on its own and in combination, because a fix in one can break another.
Release process
How the release gate decides
Illustrative
Grey dots are repeated runs of the same version, which vary on their own. The shaded band is their normal range. A pull request that scores inside it passes. One that falls clearly below it is blocked.
- 07
Make the gate trustworthy
A naive gate treats every dip as a regression and gets ignored. Each baseline run is stored as a cohort of repeated runs. The gate pools the variation of the last 20 cohorts per test pack and blocks a pull request only when a pack falls more than two standard deviations below its baseline. Issues users once raised cannot quietly come back.
- 08
Set the lifecycle
We turned these pieces into the six-stage lifecycle. Every skill and subagent declares a version and changelog, and CI checks them. Each eval result is tied to the version and commit.
Online evals and data safety
What gets scored in production
Each bar is the share of conversations of that type that get scored. Conversations with a strong signal are always scored, routine ones are sampled.
- 09
Watch quality in production
We designed and built monitoring around the signals users leave behind. A copied answer is the clearest sign that a response was useful. Negative feedback and long conversations are the clearest signs of risk. These conversations, and those with a copied answer, are always scored. Each scored conversation is checked for frustration, stuck threads, answers drifting off the question and disengagement.
Analytics
Agents360: cost against time saved
Illustrative
Time saved →
Agents360 compares agents by what they cost to run and how much time they save. Agents with high value at low cost are protected, expensive ones that pay off are optimized, and the rest are reviewed or cut.
- 10
Bring usage, cost and value into one view
Cost, quality, usage and productivity each lived in a different system. We designed Agents360 to join them on the conversation and the person. It is designed to show which subagents and skills are used, by whom and for what, what each one costs and what it returns in time saved. Every view can be split by department, office location and seniority, so leadership, QA and developers each see the decisions they own.
Business value
- 11
Measure the business value
We ran a productivity proof of concept on real conversations from consultants across seniority levels, from associates to managing directors, in the US and Europe. It showed about 2 hours saved per conversation, about €1,000 of consultant time at an average rate of about €500 per hour. The same method now runs on production conversations: each conversation is classified into one of 10 FGS workflow categories, time saved is estimated per conversation, and a short in-app survey collects ground truth.
Team practice
- 12
Take the practice beyond Fergus
Other FGS teams build their own point solutions on different platforms. We advised them on a proportionate quality loop: a few critical scenarios, clear test results and a named reviewer. The same quality loop became a shared practice across teams.
- 13
Hand over a practice, not only a system
We documented the method independently of any platform: how to define a capability, write a quality contract, choose a test surface and decide on a release.
Problems we faced along the way
Not every problem got a full fix. We solved some, and we bounded others and accepted what was left, so the product roadmap kept moving.
Setting a tolerance for normal variation
The problem: LLM results vary from run to run, so the same change can pass one run and fail the next. A strict gate raises false alarms, and a gate that cries wolf gets ignored. A gate that is too generous lets real issues through to the beta group. What we did: we measured the normal variation of each test pack from repeated runs on the main branch and set the tolerance from it, at two standard deviations below the baseline. We gate per pack, because single cases are too noisy, and borderline results warn instead of blocking. What we accepted: the tolerance is a starting value that can be tightened as run history grows, and a very small drop inside the normal range can still pass.
Knowing whether the scorers can be trusted
The problem: a scorer that gives different scores to the same input makes every result built on it meaningless. What we did: we added scorer health checks. Each scorer is tested for determinism, meaning the same input gets the same score, and for robustness to small changes in the input. Only scorers that reach the threshold on both count towards official grading. The others stay in development. What we accepted: we did not delay the gate to repair every weak scorer. Unstable ones are shown but not counted.
Deciding what a good test is
The problem: a failing test does not always mean a failing agent. Some tests asked for behavior the agent was never told to show, or lacked the context they depend on. Some questions were product decisions: when the agent asks for missing information, is that a failure or the right call? What we did: we wrote decision rules for tests. A test must check instructed behavior and carry its own context, and top priority is reserved for production-critical behavior. We separated the decision to ask from the quality of an answer. For a concrete request, the agent should make a concise attempt, so we removed a test that rewarded asking. What we accepted: the test set changes as the product does, and its owners review it.
Coverage against speed
The problem: running every test on every change was slow and noisy. A change to one skill failed on tests of unrelated skills. What we did: we tagged each test pack with the capabilities it covers, ran only the packs a change touches, and kept a small priority set that still covers every production skill. Changes to the orchestrator touch everything, so they run more repetitions. Beta and staging releases always run the baseline set. What we accepted: not every case runs on every change.
Proving value, not only quality
The problem: a faster answer is not a productivity gain unless the result is usable, and time saved cannot be read from a log. What we did: we counted only conversations with a usable result, priced the time at the rate of the person who would have done the work, and checked the estimate against a short in-app survey. What we accepted: the figure is an estimate of valued time, not a timesheet and not cash.
Closing the gap between tests and real use
The problem: offline tests only cover what we thought of in advance. Real conversations differ. What we did: we built the benchmark from issues users raised, scored live conversations on signals such as negative feedback and copied answers, and turned strong conversations into new test cases. What we accepted: scoring in production is sampled rather than complete, so scoring cost goes where the signal is.
What we built
Offline evals
Offline evaluation system
Versioned test packs, a parallel runner, deterministic and LLM-based scorers, and stored evidence for every case: response, tools used, trace and cost.
Memory evals
Coverage for five memory layers, tested separately and together.
Release process
Agent development lifecycle
Six stages from stakeholder alignment to monitoring, with roles, outputs and release decision rules.
Reliability-aware release gate
A normal range per test pack, from run history. Blocks confirmed regressions, warns on borderline results, and flags flaky tests separately.
Online evals and data safety
Production monitoring
Designed and built: signal tracing, priority sampling, four scorers combined into one weighted score, graded alerts and mining of strong conversations into new test cases.
Anonymization service
LLM-based removal of personal data from production conversations, which caught every personal data item in its validation test set.
Analytics
Eval analytics dashboard
Version-against-version comparison on quality, speed and cost, trend lines, scorer health and trace inspection down to individual cases.
Agents360
Analytics design that joins usage, cost, productivity and quality per subagent and skill. Covers ROI per capability, a waste and value quadrant, and views split by department, office location, seniority and workflow type.
Business value
Productivity measurement
Conversation classification, success-adjusted time-saved estimates, in-app survey and dashboard.
Team practice
Handover library
Practice guide, maintainer guide, QA release review, operations runbook and guidance for point-solution teams.
Business outcomes
Results at a glance
Cost: one change cut the cost per prompt by 40% at the same quality. Quality: the score on a benchmark built from the issues users raised, before and after. Time: consultant time saved per conversation.
Offline evals
40% lower cost per prompt, models chosen on cost and speed
For every new model release and every major agent change, both versions ran the same eval suite side by side. One comparison caught a change that cut cost per prompt by 40% with no loss in quality. The runs also showed that switching between Claude Sonnet and Gemini Flash models had little impact on quality, because the network's structure carries much of the work. FGS can choose models on cost and speed.
Quality from 76% to 94% on the issues users raised
The benchmark is built on the issues users raised, so the score tracks what users care about. The evals showed the team where Fergus fell short and whether each fix worked. The stronger quality gate made sure fixed issues stayed fixed.
Testing is fast and affordable
Each pull request runs a focused set of priority cases instead of the full suite, and every production skill stays covered. Every eval result records its cost.
Release process
Every change follows the same path to production
Stakeholders agree on what good looks like before development starts. Changes pass evals, QA and a beta group before they reach the whole firm. Releases no longer depend on one demo or one person's judgment.
Releases follow a clear review
Each new version is compared with the previous one on quality, speed and cost. QA moved from repeating manual checks to a small, curated release pack and reviews the results weekly.
Online evals and data safety
Production quality can be watched, not assumed
The monitoring scores every conversation with negative feedback, a copied answer or a long thread, and samples the rest. Conversations without a signal are sampled at a low rate, so scoring cost goes where the signal is. Answers consultants copied and rated well become new test cases.
Real data is used safely
The anonymization service removes personal data and caught every item in its validation test set, so production data can feed monitoring, evaluation and productivity analysis.
Analytics
Agents360 is designed to show where AI creates value across the firm
Agents360 is designed to give the team one structure for seeing how the agent network is used. Usage per subagent and skill shows which capabilities carry real load and which are expensive but rarely used. ROI per capability sets cost against time saved and gives a clear cut, protect or optimize list. The geographic view shows which capabilities each office relies on, so rollout and support can follow real demand. The seniority view shows how agents are used across the firm, from associates to managing directors, and where adoption needs support.
Business value
About 2 hours saved per conversation
The proof of concept found it, and the same method now runs on production conversations. The average is taken over conversations that produced a usable result, not over all conversations. Saved time is priced at the rate of the person who would have done the work. At an average consultant rate of about €500 per hour, that is about €1,000 of consultant time per conversation. This is valued time, not cash saved. On fixed-price work, that time improves margin for the same billable hours.
Team practice
The quality loop is shared across the firm
Developers own the evals for what they build. QA has a clear release review. Leadership uses the dashboards to decide. Teams outside Fergus apply the same loop.
Key numbers
What we learned
LLM judges need instructions too
Out of the box they are too literal. Calibrating them is part of the work, not a detail.
Measure the noise before you gate on it
A gate that knows its own variance gets trusted. One that cries wolf gets ignored.
Fewer, deeper tests beat many shallow ones
A focused baseline with clear intent catches more and costs less.
Efficiency without efficacy is not a productivity gain
Time saved only counts when the conversation produced something usable.
Start small outside the main platform
A point solution needs a few critical scenarios and a named reviewer, not a copy of the full framework.
Quality is a team practice
Evals work when developers own them, domain experts define what good looks like, and leadership uses the results to decide.
Check the test before you blame the agent
Some failures come from tests that are wrong. Audit failed cases first.
Not every problem needs a full fix
Bound it, keep watching it and keep shipping. Time goes where it removes the most risk.
Working on something similar?
Tell us about the workflow. The first meeting is free.
Next case study →
Sourcera: An AI engine that finds savings in indirect spend and drafts the tender.