Scorers

An agent finishing a job tells you nothing about whether it did it well. A scorer is the thing that grades the finished work, so quality becomes a number you can watch over time instead of a feeling you get from spot-checks.

A table of scorers, each with a lane of Fast or Slow, a sampling setting such as "20% of runs", and the agents it grades. One row is marked "Unused".

Fast and slow

Every scorer sits in one of two lanes, and the lane is a trade.

Fast scorers are cheap enough to run constantly. They catch the obvious: an empty answer, a wrong shape, something that plainly failed.

Slow scorers are the careful ones — the judgement calls. They cost more and take longer, which is exactly why you would not run one on everything.

How often each one runs

That is what sampling is for. A scorer can grade every run, a share of runs, or never.

Grading a fraction is usually right for a slow scorer: you are trying to see a trend, and a fifth of the runs shows you a trend perfectly well at a fifth of the cost. Grading everything is right when a single bad result matters on its own.

A scorer set to never stays declared but does no grading — the way to mothball one without deleting it.

Some scorers are marked reference only. They are not grading current work at all; they exist as a yardstick to compare against.

A scorer marked Unused is not attached to any agent. It runs on nothing. This is the most common reason a scorer you set up appears to do nothing at all.

Where to go next

• Agents — the workers whose output gets graded.
• Scenarios — checking that a journey works, as opposed to grading how well it went.
• Virtual customers — pretend users who go looking for problems a written test would not.