How autoresearch works, simplified. it's a pattern that lets AI agents run experiments and improve anything…
The play
Set up an AI agent loop that runs automated experiments on a measurable target, keeping improvements and reverting failures.
How it works
- Create program.md defining the agent's goal, rules, and constraints
- Create train.py as the single file the agent can modify
- Create prepare.py to measure results without agent access
- Run the agent loop to test experiments and keep scoring improvements
- Select a metric where better is objective and measurable
Result reported by Shann³
Shopify achieved 53% faster Liquid engine parsing. Portfolio load time dropped from 50ms to 25ms in 4 minutes. Driveline Baseball R-squared improved from 0.44 to 0.78.
- Good for
- Teams optimizing measurable metrics in code, prompts, or performance
Try this first
Define one objective number to optimize and ensure automated evaluation with no human in the loop
Breakdown by RunAgents from Shann³'s original post, shown below.
The original post
Shann³
@shannholmberg
how autoresearch works, simplified
it's a pattern that lets AI agents run experiments and improve anything you can measure
three files is all you need, everyone should be running it. ↓
> program. md is where you tell the agent what to do. your goal, the rules it has to follow, and any constraints. think of it as the job description
> train. py is the only file the agent can touch. this could be code, a config, a prompt, a math equation, whatever you want optimized
> prepare. py is the scorecard. it measures results and the agent can never edit it. if it could, it would just fake better scores
the loop it uses:
1. agent reads your goal
2. tries an experiment
3. measures the result
4. keeps it if the score improves, reverts if it doesn't
5. repeats for as long as it's improving.
it can run 100+ experiments.
a common conception is that it's for ML, but it can be applied widely.
if you can score it, you can autoresearch it
> Shopify ran it on their Liquid engine. 53% faster parsing from 93 automated commits
> someone pointed it at a portfolio website and load time dropped from 50ms to 25ms in 4 minutes
> Driveline Baseball used it for pitch velocity prediction. R-squared went from 0.44 to 0.78
marketing, trading strategies, prompt engineering, code performance.
we have three conditions for it to work:
> one number to optimize
> automated evaluation with no human in the loop
> one file the agent can change
anything where "better" is subjective doesn´t really work. brand design, UX, pricing without user traffic data, so skip that.
the edge here is picking the right metric
give it a bad one and it will confidently optimize the wrong thing
511 likes · 64 reposts·18d