inference

diff - training vs inference

Backlinks

diff - training vs inference

from link not tracked... you need only a single link not tracked for link not tracked on a sequence to sequence comparison. During inference, however, you need a forward pass per link not tracked

see in context
video - Code Generation and Maintenance at Scale with Agentic Workflows - 2024-10

At 14:00 he talks about the problem of compounding errors and how you can get in a bad state that you can't recover from. The solve is to use firecracker to be able to snapshot the memory of a process at a known good state and then try 10 hypothetical changes, then you rely on things like unit tests, etc. to determine which ones are the best and you establish a quorum to decide what should become the new base to fork from x concept - automatically testing AI generated code concept - deciding which branch is the best
...
This also reminds be of neuro-symbolic approaches to reasoning using the abduction methods. Basically concept - generating hypotheses hypothesis then winnowing things down by hypothesis testing. The difference here is that these seem to be short-term hypotheses and branching with the aim of getting to some new known good checkpointing, then forking off of that to go further. That feels a little like beam search. This is all to avoid the compounding error (AKA error accumulation), which is particularly important because of dim - reliability -- low and a lack of revisiting ground truth. LLMs doing multi-step inference tend to have exposure bias and distribution shift, which causes the errors to get worse over time.

see in context