eval dataset

evals datasets

link not tracked

Backlinks

blog post - the three pillars of AI observability - 2025-11

Once you capture annotations, you should save them into eval datasets which are the basis for offline evals. Each time you want to make a change to your application, you should evaluate it against the datasets you've accumulated to approximate its impact. Although people often use the term golden dataset, we have seen a shift away from this concept in favor of a more fluid approach called "dataset reconciliation", where the goal shifts to incrementally and frequently updating datasets to represent real-world behaviors, rather than commissioning one up front.
...
saving annotations into your eval dataset
...
A move away from golden-set to dataset reconciliation. diff - golden set vs. dataset reconciliation

see in context

We've built annotation tools directly into the trace viewer so product managers, domain experts, and other stakeholders can review traces, flag issues, correct outputs, and build datasets without getting lost in JSON or touching code. These annotations flow directly into your evaluation eval datasets, creating a continuous improvement loop that leverages expertise across your entire team.
...
evals persona -- product manager

see in context
saving annotations into your eval dataset

saving annotations into your eval dataset

see in context