⇩ Markdown

considering dark software factories→considering dark software factories -- changelog→mechanisms for making agents more reliable→LLM as a judge

LLM as a judge

Similar to LLM as a verifier

example of using LLM as a judge for message structure

Backlinks

LLM as a verifier

Similar to LLM as a judge

see in context
agentic code reviewer

A bit like LLM as a judge

see in context
example of using LLM as a judge for message structure

Here is an example of using LLM as a judge for message structure. This seems like a good example of having LLM judges check for discrete signals, because it doesn't ask for anything generic. It asks for three specific things.

see in context
good examples of LLM as a judge

good examples of LLM as a judge

see in context
having LLM judges check for discrete signals

^ LLM as a judge can check for discrete signals

see in context
mechanisms for making agents more reliable

LLM as a judge (subjective evaluations)

see in context
tweet - llm as a judge can be crazy expensive

LLM as a judge dim - costs -- high link not tracked so the LLM model in LLM as a judge is often smaller than the task model

see in context
the LLM model in LLM as a judge is often smaller than the task model

often you will use a link not tracked to check the output of the bigger one
...
LLM as a judge link not tracked

see in context
why give evals a different name than just tests?

Unlike most traditional tests, you need to decide the level of exactness that you care about. Often there is more than one acceptable answer. Or worse, the answers are not eval type -- verifiable, so you need to get LLM as a judge involved.
...
dim - comparison exactness

see in context

If you do use LLM as a judge you need to do fitness to task optimization for the judge as well. For example the LLM model in LLM as a judge is often smaller than the task model because tweet - llm as a judge can be crazy expensive, otherwise. So, you need to experiment to see just how small that evaluation model can be and still get evals you can trust. In traditional testing, you primarily test the code.

see in context