“How to Run Better Research With Agent Fleets” by Joshua Fonseca Rivera

In the current safety landscape, researchers are encouraged to use as many agents as possible, even in “fleets,” where a large number of them go out and run experiments while you, uh... wait for them to come back? I've spent the last few months of the Anthropic Fellows Program trying this out. My first attempts were naive, but after a few iterations I found an approach that works well for me. Research is a double loop I want to propose a mental model of research as a double loop, an idea that builds on double-loop learning (Argyris, 1977). The outer loop is reserved for the inputs we currently refer to as "human taste": choosing which questions are worth answering, designing a trustworthy measure, and deciding which direction to go. Agents can help with this, but they’re still bad at making these decisions autonomously. The inner loop turns each idea you test (an arm as claudius likes to say) into a comparable result. An example of a very simple inner loop might be: Write a system prompt. Run the model with that prompt on a set of test questions. Score the model's replies. Compare the scores against a control [...] --- Outline: (00:33) Research is a double loop (02:15) What happens when you don't operationalize (05:21) How to set up a fleet (05:31) 1. Freeze everything but the arm (06:17) 2. Brief each agent (06:33) 3. Protect the pipeline (07:02) 4. Read the raw outputs yourself (07:22) 5. Add a manager agent (08:03) What a fleet run looks like --- First published: November 6th, 2026 Source: https://www.lesswrong.com/posts/WKtD59uHLKwEYGFCm/how-to-run-better-research-with-agent-fleets --- Narrated by TYPE III AUDIO . --- Images from the article: <a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/v1793961866/lexical_client_upload