regression test suite

Backlinks

maybe some evals should fail in order to be useful?

I'm less sure if that's always applicable. I think there is utility in evals functioning like a regression test suite, for example to support things like link not tracked or link not tracked.
...
edit: 2026-01-27 - In fact, in link not tracked, they call out explicitly having different set types, with one of them being link not tracked

see in context
mechanisms for making agents more reliable

backstops (deterministic, and rule-based) like linting, regression test suite

see in context
regression monitor

See also regression test suite

see in context
why give evals a different name than just tests?

More closely tied to experimentation than traditional tests. Or at least more like TDD than a regression test suite. For example, they can be directly tweakable by evals persona -- domain experts in integrated prompting environment, and evals can be co-evolved with prompts. That need for tweaking makes the need for manually triggered in the app context be important.

see in context