hypothesis testing

Backlinks

chat - abduction

In artificial intelligence, abduction is used to create systems that can reason about incomplete or uncertain information by generating hypotheses that explain the available data. This is particularly useful in fields like diagnostic systems, natural language understanding, and knowledge representation. X abduction concept - generating hypotheses hypothesis hypothesis testing

see in context
video - Code Generation and Maintenance at Scale with Agentic Workflows - 2024-10

This video explores one solution to that problem, a product called grit, and goes into the technical details about how it uses massively concurrent forking agentic workflows to solve the problem. I found the section at 14:00 about the use of firecracker to do fast, exploratory hypothesis testing (which feels a little like beam search) to be particularly intriguing.

see in context

At 14:00 he talks about the problem of compounding errors and how you can get in a bad state that you can't recover from. The solve is to use firecracker to be able to snapshot the memory of a process at a known good state and then try 10 hypothetical changes, then you rely on things like unit tests, etc. to determine which ones are the best and you establish a quorum to decide what should become the new base to fork from x concept - automatically testing AI generated code concept - deciding which branch is the best
...
This also reminds be of neuro-symbolic approaches to reasoning using the abduction methods. Basically concept - generating hypotheses hypothesis then winnowing things down by hypothesis testing. The difference here is that these seem to be short-term hypotheses and branching with the aim of getting to some new known good checkpointing, then forking off of that to go further. That feels a little like beam search. This is all to avoid the compounding error (AKA error accumulation), which is particularly important because of dim - reliability -- low and a lack of revisiting ground truth. LLMs doing multi-step inference tend to have exposure bias and distribution shift, which causes the errors to get worse over time.

see in context