AI Compass / Knowledge cards

Self-improving loop with locked checks

Checked 17 October 2026 · Michael Engel · Automation · Prompting

In March 2026, Andrej Karpathy showed with AutoResearch how an agent runs experiments on its own overnight: change code, measure, keep improvements, discard the rest. According to the README, the file with the evaluation stays unchanged, and the agent edits only a single file. AI LABS transfers this to Claude Code. The checks, that is, the check rules, are locked, and only the working instructions may change. Otherwise what often happens with optimisers happens: the agent improves the measurement instead of the result.

What it is for

A recurring task gets better with every round without you sitting next to it. The yardstick stays yours.

Steps

1. Check whether the task fits. According to the video, four conditions must hold: the task comes up often, your quota is sufficient, the result is clearly measurable, and the agent can run it itself. Only one feature per loop, never a whole app.
2. Capture the project knowledge in a skill so that every run starts from the same state. In the video it is called project-context.
3. Have the checks written and go through them with the agent until you approve each one.
4. Move the checks into a folder of their own, lock changes there and commit everything. In Claude Code, the lock works with a deny rule in the settings, for example "deny": ["Edit(/checks/**)"]. The syntax comes from the docs, not from the video.
5. A build agent with fresh context works with two files: a results file and the working instructions program.md.
6. After each round, the self-improvement may only rewrite the “How to work” section in program.md, never the checks.
7. Set a budget or round limit (my own assessment). At the end, check what the checks do not cover: in the video, all 11 checks were green, yet the order form had not been tested at all.

Limits

• The lock in the settings is not a watertight wall. According to the docs, deny rules apply to Claude’s file tools, to known commands such as sed or tee and to redirects such as > file. A script that opens files itself, for example in Python, bypasses them. For a lock at operating-system level, the docs name the sandbox. git helps too: every change to the checks shows up in the diff.
• Quota: On the $20 plan, the limit is reached before a long loop finishes, according to the video.
• Difference from the original: at Karpathy, according to the README, the human writes the program.md and the agent only changes the code. In the video, the agent also writes its own instructions. That saves you work, but you see less often what has changed. So track changes to program.md with git (my own assessment).
• The example figures in the video are the creator’s claims.
• The finished skills from the video exist only in the paid community. The pattern can be built yourself.

Links

• “Done” only with evidence: what checks look like as check commands with expected output.
• Let Claude work until the goal is met – with /goal: the built-in loop for a single task.
• Separate building and checking: a critic with fresh context instead of fixed checks.

Sources

• AI LABS: He Finally 10x Claude Code With This Method, 2 October 2026, 2:32–6:29, 8:09, 8:42–11:56. Sponsored by Upstash Box (approx. 6:29–7:35), the finished skills are in the paid community
• Andrej Karpathy: autoresearch, README, retrieved 17 October 2026
• Claude Code docs: Permissions (deny rules for Edit, scope), retrieved 17 October 2026

Status

Checked 17 October 2026 · Maturity new (pattern available, basis AutoResearch is public; results in the video only from the creator)

Want to try this with your team? That’s what the AI Lab is for. See the AI Lab