⇩ Markdown

considering dark software factories→considering dark software factories -- working area→weird names can be helpful→AI unit→compound system

compound system

concept - The compound system is exposed by API are different from those exposed via application

Backlinks

AI unit

I like avoiding model because these days (early 2026) most of these units we interact with are compound systems like AI unit -- GPT-5.

see in context

I'm also reticent to call them agents because I believe agents are perhaps a sub-categorization of these AI units. They are, specifically, a more complex compound system of more basic types of AI units. For example, people would call something like Claude Code an agent because it is an agent harness. So, I'd like to reserve agent for more complex compound systems.

see in context
blog post - the three pillars of AI observability - 2025-11

evals: Quantify performance, both in production (online) and in dev and CI (offline), to understand how well your application works and systematically improve it
...
dim - model performance (but not really model performance... more like compound system performance)

see in context
concept - The compound system is exposed by API are different from those exposed via application

compound system exposed via link not tracked Can be different from the exposed via link not tracked. For example company - Anthropic artifacts are only exposed through the app. link not tracked is also only part of an app experience.

see in context
fitness to task optimization

link not tracked

Could be link not tracked, or could be compound system optimization of a broader sort.

see in context
model call

part of a compound system

see in context
why give evals a different name than just tests?

evals are a bit like unit or system tests, so why not just call evals "tests" like we have for every other software system we've been evaluating for decades? Maybe because of how they started? Evals started off simply evaluating models and only over time have companies started to develop more complex compound systems, including agents, where evaluating the system is more like doing a system test.

see in context

The thing you're testing, though it is closely tied to AI functionality, is likely a compound system and the evals, if they're not pass-fail, may be differentiable and could act as a sort of loss function on that compound system.

see in context