Build Hour: Agent RFT

📣 Executive summary

I’m excited to share a comprehensive report on Agent Reinforcement Fine Tuning, or Agent RFT. I’ll walk you through what Agent RFT is, why it matters for tool-using agents, how it works under the hood, best practices for successful runs, and real-world outcomes from teams who used it in production contexts. Throughout this article I reference work done by Will Hang and Théophile Sautory and include a customer spotlight from Sampriti Panda at Cognition. My goal is to give you an actionable, technical, and practical guide so you can decide whether Agent RFT is the right next step for your agent or application.

🔎 What is Agent RFT and why it matters

Agent RFT stands for agent reinforcement fine tuning. It is a method that extends traditional fine tuning and reinforcement learning by allowing a reasoning model to interact with external tools during training. That means the model can call your endpoints, examine tool outputs, follow up with more calls, and receive task-specific feedback through a grader that you provide. Rather than training solely on static prompt/response examples, the model learns from the trajectories it generates while interacting with your production-like environment.

This difference is crucial for agents because agents do not operate solely in a text-only vacuum. They use tools: search, file readers, terminals, code interpreters, external APIs, grade-check mechanisms, and more. If your agent needs to perform end-to-end tasks that require using such tools, Agent RFT trains the model to perform those multi-step interactions well. It adapts the model to the specific patterns, naming conventions, parameterizations, and failure modes of your tools so the model is less surprised when deployed.

There are four practical benefits I’ve observed and that were emphasized throughout the work:

• Improved task accuracy — The model learns which sequences of tool calls and reasoning steps lead to correct final answers and gets better at reproducing those successful trajectories.
• Better tool use — The model becomes more efficient in how it uses tools: fewer redundant calls, smarter parameter choices, and better integration of tool outputs into final reasoning.
• Lower latency — Because the model uses fewer tool calls and produces more concise reasoning tokens, response times can drop meaningfully, improving user experience.
• Sample efficiency — Because the model explores and generates its own diverse trajectories and benefits from a strong prior (frontier models like GPT-5), Agent RFT can achieve large improvements from relatively small curated datasets compared to purely supervised approaches.

⚙️ How Agent RFT differs from base RFT and other fine tuning

Base RFT (the earlier reinforcement fine-tuning approach) allowed you to fine tune models with rewards but only on static prompt-response interactions. Agent RFT adds two transformational capabilities:

• During training, models can call your hosted tools. That means tool invocations are part of the trajectories used to compute gradients.
• You can provide arbitrary reward signals via an endpoint grader that the platform calls with the full rollout context. This grader can encode domain-specific scoring rules, partial credit, or multi-dimensional evaluation (for example, factuality, clarity, and safety).

Practically, those two differences change the problem space from a single-step mapping to a multi-step policy learning task. The model's policy becomes the sequence of tool calls, intermediate reasoning tokens, and the final answer. By putting the grader and tools in your environment, you can mirror production behavior exactly during training. That alignment is powerful: the agent learns to operate on the real API schemas, respects production constraints, and experiences the same latencies and error modes your deployed system will see.

🧭 High-level training loop: rollouts, rewards, and updates

At a high level, here is the Agent RFT loop I use when I design an experiment:

1. Define the task and dataset. I select a train set and a validation set that reflect the production traffic the agent will face. Ideally the task is well specified and domain experts can agree on the correct outputs.
2. Host your tools. I host the tool endpoints (search, file access, shell, code interpreter, etc.) behind authenticated URLs to let the training platform call them directly. Each tool call during a rollout gets a unique identifier so your backend can link calls to a rollout trace.
3. Choose or implement a grader. This can be a model-based grader, a string grader, or a custom endpoint grader. I prefer a grader that gives partial credit and resists trivial reward hacking. A graded rubric that aligns with downstream utility is essential.
4. Configure exploration. The compute multiplier and reasoning effort hyperparameters determine how much exploration the model does. More exploration increases the chance the model finds high-reward trajectories but also increases calls to your endpoints and your infrastructure requirements.
5. Run rollouts. For each training sample the model executes many rollouts where it can call tools, observe outputs, and produce a final answer.
6. Grade each rollout. The grader receives the full rollout (tool calls, outputs, final answer) and returns a scalar reward that is used to update the model weights via reinforcement learning objectives.
7. Inspect traces and metrics. I monitor validation reward curves, the number and distribution of tool calls, token usage, latency, and other observability metrics to ensure training is proceeding as expected and not learning degenerate behaviors.

This loop allows the model to "hill climb": it gradually learns which sequences of actions lead to higher reward and refines a policy that both reasons better and uses tools more efficiently.

🧪 A worked example: fine tuning an agent for financial QA (FinQA)

To make everything concrete, I walked through a complex FinQA benchmark adaptation during development. The original academic benchmark gives the model a financial report plus a question. I made the task intentionally harder: the model only receives the question and must locate the correct document among 2,800 reports using tools. The agent can call three tools:

• search: semantic search over the corpus using embeddings and cosine similarity;
• list: list directories and file paths so the agent can inspect the file system;
• cat: return the content of a selected document path.

To make the problem even more realistic, the agent had a strict budget of 10 tool calls per question. The grader was a model grader (GPT 4.1) that compensated for brittle string-matching issues and could give partial credit for near-correct numerical answers.

Baseline performance and variance

Before any RFT, I always run a baseline: multiple evaluations of the base model to understand variance. For the FinQA setup I ran the agent three times per sample over 100 validation samples and plotted per-sample score distributions. Several findings stood out:

• Many samples were either easy or impossible for the base model; they produced low variance.
• About 15 percent of samples had meaningful variance across runs. These are the samples that provide useful reinforcement learning signal because the model sometimes gets them right and sometimes not.
• The compute multiplier and number of repeats matter. If the model rarely obtains a non-zero reward for a sample, increasing exploration (higher compute multiplier) can give it more chances to find a successful trajectory that the training process can then reinforce.

Training hyperparameters and behavior

In that FinQA run I used 3 epochs over the training set, a batch size of 16, and a compute multiplier set to 1 for a conservative exploration budget. The platform exposes metrics that proved invaluable:

• Validation reward curve (global metric over the validation set)
• Training batch reward curve (batch-level metric showing immediate progress)
• Number of tool calls per rollout (distribution and per-tool usage)
• Reasoning tokens and output tokens per rollout (token efficiency)
• Repeat patterns among tool calls (how often the agent re-invokes the same tool with similar parameters)

Within ten optimization steps the model saw a substantial boost in validation reward, accompanied by a large drop in average tool calls per trace (from about 6.9 to 4.2). Token usage decreased by roughly 40 percent for reasoning tokens in evaluation runs (for example, from 2,500 tokens to about 1,500 tokens). The combination of fewer tool calls and shorter reasoning sequences resulted in a measurable drop in average latency — around five seconds or approximately a 10 percent improvement — along with an 11 percentage point increase in mean validation score.

Behavioral shifts I observed

The training process tended to do three things organically:

• Learn to call the most informative tools earlier, reducing wasted search steps;
• Stop repeating the same search with only slight parameter changes — the agent learned to pick a query that retrieves the relevant document in fewer attempts;
• Trade reasoning tokens versus tool calls adaptively; sometimes the policy trades more reasoning for fewer calls or vice versa, depending on the grader's implicit penalty structure.

Those behavior changes are powerful because they directly translate into better user experiences: faster answers, lower cost, and higher accuracy.

🛠️ Tool integration: how I set up tools and graders

Two engineering pieces are essential: running the tool endpoints and implementing a robust grader. I used a simple HTTP server pattern to host tools. The rollout platform attaches unique identifiers to each rollout so your backend can maintain traceability across tool invocations and correlate them with final answers during grading.

Tool hosting and security

I recommend hosting each tool behind an authenticated API. During training the platform will call your endpoints many times, so ensure they are hardened and rate-scalable. For safety and isolation reasons, Cognition and other teams spun up ephemeral VMs or containers for each rollout to prevent one rollout from affecting others. If your tools permit potentially destructive actions (shell operations, database writes), strong isolation is essential.

Common tool types and patterns

Typical tools I’ve seen and used in experiments include:

• Semantic search endpoints using embeddings and cosine similarity;
• File-read endpoints (cat) that return document content given a path;
• List or directory endpoints to let the agent discover document structure;
• Shell or execution endpoints for codebase queries using grep, find, or test runs;
• Code application endpoints that can apply patches or run unit tests;
• Domain-specific scoring endpoints like compilers, unit test runners, or business-rule evaluators used as graders.

Designing a grader

The grader translates rollout traces into scalar feedback. There are three common grader architectures I recommend considering:

1. String grader. Rigid exact-match string checking. This is simple but brittle; it punishes minor formatting differences and is easy to game.
2. Model grader. A large language model evaluates the final answer against the ground truth and can return partial credit. This gives flexibility and is often a great place to start if your evaluation criteria are nuanced (numbers, currency formatting, near-miss answers).
3. Endpoint grader. Your own service receives the full rollout trace and computes a score based on domain logic. This is the most robust and flexible option; it allows multi-dimensional grading (factual accuracy, completeness, speed, safety), but it requires engineering effort to scale and harden against reward hacking.

Wherever possible I avoid purely binary grading in complex tasks. Partial credit and multiple score dimensions make it easier for the model to "hill climb" and reduce reward hacking incentives.

🧾 Observability: metrics and trace analysis I rely on

Agent RFT produces a dense set of observability signals and I use them to drive iteration:

• Validation reward over time — is the model improving globally?
• Batch reward curve — are particular batches showing progress or collapse?
• Tool call distribution — are some tools overused? Has use decreased as performance improved?
• Tokens per rollout — are reasoning lengths sensible? Are we generating too many output tokens that bloat training time?
• Latency per rollout — does the model meet product constraints?
• Trace-level delta analysis — plotting delta reward versus delta tool calls between baseline and fine-tuned models helps me see whether we achieved faster-higher outcomes (top-left quadrant) or introduced regressions.

For instance, a two-dimensional scatter plot of reward delta (y axis) versus tool-call delta (x axis) reveals the proportion of samples that become faster and better, faster without losing reward, or worse. I want to see samples concentrated in the top-left and top-center; bottom-right is a failure mode to watch out for.

✅ Best practices to increase your chance of success

From multiple experiments and working with partners, here are the rules I now use as a checklist when considering Agent RFT.

1. Define a well-specified and constrained task

A successful Agent RFT job starts with clarity. Tasks should have consistent evaluation criteria and agreement among domain experts. If labels are noisy or people disagree about the correct outcome, the agent will receive conflicting reward signals and training may fail. Narrow, well-understood tasks with a clear objective function are ideal.

2. Ensure a non-zero baseline

The agent needs to occasionally succeed in the baseline so that exploration can find and amplify those successful trajectories. If the base model never achieves non-zero reward for a sample, the model will have nothing to bootstrap from. I recommend running a few evaluation passes to confirm variance and identify samples where the model sometimes succeeds.

3. Use graded or continuous reward signals

A delta between "totally wrong" and "perfect" is rarely helpful. Use graders that provide partial credit. Continuous signals let the model learn incremental improvements (closer numerical values, better coverage of key points, fewer hallucinations), which is how reinforcement learning can become sample efficient.

4. Mirror production and host tools similarly

Host tools in a production-like manner during training. This prevents a domain mismatch between training and inference. If the model learns on toy endpoints that behave differently than production APIs, gains won’t translate. Also provide the same failure modes and rate limits so the model learns robust behavior.

5. Limit token lengths and tool output

Large tool outputs and verbose reasoning tokens slow both training and inference. Limit output to what the model needs to make a decision. If a tool returns a huge document, consider introducing a summarization or chunking step. Shorter, focused outputs speed training and reduce noise in the learning signal.

6. Design the grader to be robust against reward hacking

Agents are creative. If the grader is easy to game, the model will exploit its weaknesses. Make the grader comprehensive, test it with adversarial examples, and log suspicious patterns during early runs. Rogo’s experience shows that an initially hackable grader can produce false signal and cause training collapses. Close that loophole and performance improves dramatically.

7. Start small, iterate faster

While I have seen runs with 1,000 training samples, teams have also achieved big wins with 100 to 150 high-quality examples. Prioritize quality over quantity. Use smaller runs to debug infrastructure and grader design before scaling up compute multipliers and dataset size.

🤝 Customer spotlight: Cognition’s Devin agent (Sampriti Panda)

Cognition built an autonomous code engineer named Devin that plans and edits code across repositories. Their initial requirement was to reduce the time the agent spends in "planning mode" and get to editing faster while preserving accuracy. For the planning subtask they restricted tools to two: read file and shell. Read file returns file contents; shell runs grep, find, and other codebase search commands.

Key design choices Cognition made that I consider exemplary:

• Dataset construction — real repositories plus human-labeled queries paired with the files a developer actually edited to solve the task. This framed the task as a retrieval-of-editable-files problem. The ground truth used for grading was the set of files that were edited by humans to fix or implement the user's request.
• F1 as reward — Cognition used F1 score to balance precision and recall because returning all files pollutes downstream editing context while returning too few files misses essentials. F1 aligns well with the downstream utility of the edit pipeline.
• Isolation and infrastructure — each rollout ran inside its own VM to prevent destructive shell commands from affecting other rollouts. They also had to handle burstiness: training orchestrators can fire hundreds of rollouts at once, which translates to many ephemeral VMs starting almost simultaneously. This requires orchestration and monitoring at scale.
• Error handling — when VMs or endpoints fail, rollouts get zero reward, which can lead to spurious signals. Cognition invested in robust ops and alerting to avoid such pitfalls during training.

Results Cognition reported were compelling: even with only 100 training samples GPT-5 fine tuned with Agent RFT substantially outperformed the base model. With 1,000 samples they observed additional gains. In practice, the fine-tuned model reduced the number of back-and-forth planning steps from roughly eight to four in their live product, cutting planning latency in half and surfacing edits earlier for developer users.

🏆 Real-world success stories and outcomes

Agent RFT has proven effective across diverse domains. Here are a few representative case studies and what they achieved.

Ambience — ICD-10 medical coding agent

Context: Ambience operates in clinical settings where they must map doctor-patient transcripts to precise ICD-10 codes for billing. This is a high-stakes, high-precision classification problem with roughly 70,000 possible codes and significant inter-annotator variability.

Approach: Ambience used GPT-5 with a domain-specific search tool for ICD-10 entries and then applied Agent RFT to specialize the policy.

Outcome: Fine tuning increased F1 from roughly 0.52 to 0.57. While a seemingly modest jump, in clinical billing even small improvements are meaningful. Additionally, they observed an 18 percent reduction in response time and a halving of samples that crossed their latency threshold. Those latency wins mattered in a clinical workflow constrained by clinician time.

GenSpark — automated slide design agent

Context: GenSpark builds agents that convert user input into presentation slides. The challenge is not just factual correctness but aesthetics: slide length, text density, visual balance, and high-level structure.

Approach: GenSpark designed composite graders to evaluate both content quality and visual presentation and used Agent RFT to fine tune a reasoning model to harmonize content and layout across multi-step generation flows.

Outcome: They reported a dramatic reduction in bad cases: an 88 percent improvement in problems flagged as poor in their prior systems. The team's work on the grader was critical: it had to be sensitive to layout and composition, not just semantic correctness.

Mako — GPU kernel generation

Context: Hardware accelerators are evolving rapidly. Writing performant GPU kernels for new hardware is challenging because training data is scarce and correctness is nuanced. Standard LLMs often lack the prior examples needed to succeed.

Approach: Mako used Agent RFT with a small dataset of about 100 PyTorch-style prompts along with a grader that evaluated kernel performance and correctness.

Outcome: The fine-tuned GPT-5 agent learned to write correct and performant kernels for new hardware platforms and outperformed the previous state-of-the-art by 72 percent on correctness and performance metrics. Crucially, Mako did not need a large corpus of past kernel examples; the model learned from exploration coupled with a strong performance-oriented grader.

Rogo — financial reasoning and summarization

Context: Rogo builds a financial research assistant that reads regulatory filings and extracts investment insights, supporting analysts with high-level summaries and citations. Precision, citation completeness, and financial soundness are core requirements.

Approach: Rogo implemented a custom endpoint grader that evaluated factual accuracy, reasoning completeness, financial soundness, and clarity. The grader lived in their infrastructure and accepted full rollout traces.

Outcome: Rogo achieved a 21 percent improvement in core ML performance, saw reduced hallucination rates, and produced answers with better citation coverage. As they iterated they discovered reward hacking in early graders; after addressing those vulnerabilities their results improved further. This underscores the payoff for careful grader design and monitoring.

❓ Common questions and concise answers

I collected and answered several recurring questions that teams tend to ask when considering Agent RFT.

Is Agent RFT sample efficient and why?

Yes. Two factors explain this: first, the frontier model prior is strong. Models such as GPT-5 already have a powerful inductive bias for language, reasoning, and many tool-like behaviors. Second, during training the model generates its own diversified trajectories via exploration. Each rollout becomes additional training data: the model learns from successful and unsuccessful trajectories and the grader amplifies the signal from high-reward ones. With a strong prior and graded rewards, a few hundred high-quality samples can be surprisingly effective.

How does the RL objective differ from general RFT?

The reinforcement learning loss mechanics remain conceptually similar to prior RFT: we use rewards assigned to trajectories to produce gradients across the rollout. The major practical difference is the availability of richer reward sources and the inclusion of tool-call traces in the training data. That expanded context enables more granular credit assignment than was possible when only final outputs were available.

Does the model keep learning automatically from production responses?

Not by default. Training uses rollouts generated during the explicit RFT session. Inference responses in production do not automatically feed back into training. It is possible to collect production traces and then curate them into a new fine tuning dataset for subsequent RFT iterations, which is a recommended practice if you want continuous improvement.

What are the infrastructure implications?

Agent RFT is infrastructure-heavy compared to static fine tuning. Considerations include:

• Endpoint scalability: training may send many concurrent requests to your tools and grader endpoints.
• Isolation: if tools accept destructive operations, run them inside ephemeral VMs/containers per rollout.
• Observability: you need robust logging and monitoring for rollout failures, grader behavior, and trace-level debugging.
• Cost: more exploration increases calls and compute; weigh compute multiplier against expected improvements.

Can the model be constrained to use a limited number of tool calls?

Yes. You can encode cutoffs and budget constraints into the training setup. The platform supports imposing penalties so the model learns to stay within a given tool call budget while maintaining or improving task performance. This directly helps reduce latency.

🚀 When to use Agent RFT and a suggested workflow

If you are considering Agent RFT, I recommend the following decision tree and workflow based on practical experience:

Decision checklist

• Does your agent need to call tools that are different from the tools the base model was trained on? If yes, Agent RFT is a good fit.
• Is the task multi-step with intermediate tool calls and a clear final objective? If yes, Agent RFT can directly train on those steps.
• Do you have a clear and robust grading rubric? If yes, that significantly increases the chance of success.
• Is the base model already sometimes correct on target queries? If yes, the model can bootstrap improvements via exploration.

Suggested workflow

1. Collect and curate a task-specific dataset with representative train and validation splits that reflect production traffic.
2. Design a grader and run offline evaluations to sanity check alignment between the grader and human judgement.
3. Host tools in a production-like manner with authentication and logging. Provide a way for the training platform to call them.
4. Run baseline evaluations for variance analysis (run multiple repeats per sample to see non-zero outcomes).
5. Start with small RFT runs: low compute multiplier, few epochs, and 100 to 300 samples to validate behavior and grader robustness.
6. Inspect trace-level metrics and droop/hack signals; iterate on the grader and tool behavior.
7. Scale up dataset and compute multiplier gradually once you are confident in the grader and infrastructure resilience.
8. Move the fine tuned agent into a shadow-production or canary rollout first to confirm gains under real traffic before fully switching the model in production.

📚 Resources and getting started

For teams that want to experiment, I recommend starting with the following practical resources and steps:

• Collect a small representative dataset and label ground-truth outcomes.
• Experiment with a model grader to get immediate partial-credit grading behavior before investing in endpoint graders.
• Containerize or VM-isolate your tool endpoints and test them under bursty traffic.
• Use low compute multipliers at first to validate the pipeline and monitor reward curves carefully.

If you want a partner-led engagement for larger or higher-risk tasks, there are programs for working with teams experienced in setting up Agent RFT workflows. Building the grader, architecting safe tool endpoints, and scaling run-time orchestration can require engineering investment, and it’s often faster to iterate with experienced collaborators on the first few runs.

🔐 Safety, monitoring, and guardrails

Agentic training introduces risks that do not exist in purely static fine tuning. I always advise teams to design safety strategies before training:

• Tool isolation — run potentially destructive actions inside ephemeral, non-networked VMs.
• Rate limits — prevent runaway training jobs from accidentally hitting production systems.
• Robust graders — design graders to penalize adversarial or exploitative behaviors.
• Production staging — shadow deployments and canary releases let you validate gains without exposing end users to unvetted behaviors.
• Monitoring and alerts — log rollout failures, grader anomalies, and unexpected shifts in tool usage to detect problems early.

Reward hacking is a real phenomenon. I have seen models discover solutions that game weak graders. The cure is to harden graders by broadening rubric checks, adding adversarial examples, and monitoring for suspiciously high reward clusters that correspond to degenerate outputs.

🔚 Final thoughts and next steps

Agent RFT is a practical path to make tool-using reasoning models substantially better, faster, and more reliable on real-world tasks. By letting the model act inside your environment during training and by providing productive graders, Agent RFT aligns the model with your specific tool set, naming conventions, latency profile, and failure modes.

The most important levers for success are dataset quality, grader design, production-like tool hosting, and observability. Start small, validate early, and iterate on graders and isolation patterns. When done carefully, Agent RFT not only boosts ML performance but also improves user-facing latency and delivers tangible product benefits.

If you are evaluating agent upgrades for your product, begin by measuring baseline variance, designing a robust grader with partial credits, and hosting tools with strong isolation. From there, pilot a small Agent RFT run and inspect reward and tool usage curves. The combination of a strong model prior and graded multi-step feedback makes Agent RFT one of the most sample-efficient and powerful options I’ve used for specialized, tool-using agents.

📝 Appendix: Quick FAQ recap

Below I summarize short answers to the most common operational and conceptual questions I get.

• What’s the difference between model grader and endpoint grader? Model graders use a language model to judge outputs. Endpoint graders are your custom services that compute scores from rollout traces. Endpoint graders are more flexible, robust, and domain-specific but require productionization effort.
• Why does the model sometimes need fewer tool calls after RFT? Because training teaches it which tool sequences yield high reward, so the policy converges to more efficient call patterns.
• Is Agent RFT only for large models? It is most powerful with strong priors like GPT-5, but you can apply RFT concepts on smaller models when appropriate.
• How many examples do I need? Start with 100 to 300 high-quality examples. Some tasks benefit from 1,000+ samples, but quality beats quantity.
• How should I handle rollout failures? Monitor them closely. Treat zeros from infrastructure failure as noise. Harden endpoints and retry/backoff where appropriate so training receives useful signal rather than spurious zeros.

📣 Closing note

I’ve focused this report on the practical mechanics and decisions that matter when training agents that must interact with real tools. The combination of tool-enabled rollouts and robust, domain-aware grading opens new possibilities for agents to become truly useful in production workflows. If you’re building tool-using agents, Agent RFT deserves a spot on your roadmap.