Build Hour: Reinforcement Fine-Tuning

🔍 Introduction — What I covered with OpenAI

I recently presented at an OpenAI Build Hour titled "Reinforcement Fine-Tuning" alongside my colleagues Prashant Mital and Théophile Sautory, and with a customer spotlight featuring David Yu from Accordance. In that session I walked through why reinforcement fine-tuning (RFT) matters, how it differs from other fine-tuning approaches, and how to run practical experiments that move a reasoning model from "good" to "great" on domain-specific tasks.

In this article I report on the ideas, examples, and technical practices we discussed. I’ll explain how RFT fits within model customization, dig into an annotated demo we ran (a Eurovoc classification task), share the diagnostics you should monitor, and offer a hands-on checklist that you can apply to your own workloads. I also cover lessons from Accordance, which used RFT to improve tax and accounting workflows.

If you are building applications that require robust reasoning, policy compliance, or domain-accurate decision-making, this walkthrough will give you an actionable roadmap for getting started with RFT and for iterating toward reliable production models.

🧭 Executive summary — Why RFT and what to expect

At a high level, here's the core idea I emphasized: if your model already knows the facts (or you can surface them via prompt engineering and retrieval), and the remaining problem is "how the model thinks" (reasoning, chaining, applying rules), then reinforcement fine-tuning is the lever to pull.

RFT optimizes reasoning. Instead of training on label pairs, RFT trains with a grader — a programmatic rubric that scores candidate model outputs. The model learns to produce outputs that score higher under that rubric.
Data efficiency. You only need tens to hundreds of well-chosen examples to get meaningful improvements. RFT extracts far more signal per sample than classic supervised fine-tuning because it evaluates multiple candidate trajectories for the same prompt.
Great for domains that need strict logic. Legal policy checks, compliance pipelines, medical coding, tax strategies — any domain where reasoning matters and where you can define an objective grader — can benefit from RFT.
No massive labeled corpus required. Instead of spending months curating thousands of gold-labeled outputs, you invest time designing robust graders and representative examples.

Across the rest of this article I will unpack the tradeoffs, show the practical pipeline we used in the demo, and provide guidance for grader design, prompt engineering, debugging, and cost/latency planning.

🧠 How RFT fits into the model customization toolkit

When teams try to improve an LLM application, I encourage them to think along two axes:

What the model knows — handled by prompting, retrieval-augmented generation (RAG), and knowledge augmentation.
How the model reasons — where fine-tuning methods, and especially RFT, come into play.

If you are missing knowledge (e.g., the model lacks facts about a product catalog or a set of statutes), start with retrieval or better prompt engineering. If the model knows the underlying facts but fails to apply rules or chain reasoning steps reliably, RFT is the stronger option.

On OpenAI's platform we provide three fine-tuning techniques:

Supervised Fine-Tuning (SFT) — feed direct prompt/answer pairs. Best for format consistency, classification, and deterministic outputs.
Preference Fine-Tuning (PFT) — present examples of better and worse outputs. Useful for style, tone, or subjective preferences (marketing copy, chat personalities).
Reinforcement Fine-Tuning (RFT) — supply a grader: a programmatic rubric that can score outputs continuously. The model explores multiple reasoning trajectories, the grader scores them, and the model learns to prefer higher-scoring trajectories.

Prashant and I stressed one practical reminder repeatedly: fine-tuning is an investment. Exhaust prompt engineering and RAG first; only then reach for fine-tuning. When you do choose to fine-tune, pick the technique that aligns with the problem you are solving.

⚙️ Under the hood — Why RFT is sample efficient (Theo explains)

Théophile explained an important conceptual distinction that makes RFT efficient: each single training example yields multiple trajectories.

When RFT runs on a dataset example, the model samples many different candidate reasoning paths and complete outputs for that same input. The grader evaluates each candidate and provides a scalar score. Because the model sees multiple diverse trajectories for a single example and learns which trajectories score higher, a single example produces far more learning signal than a single SFT example does.

This is why RFT can often reach meaningful gains with only tens or hundreds of examples: you're not training on static labeled target outputs; you're training the model to climb a reward surface by exploring and receiving feedback.

🧾 Task setup — The Eurovoc classification demo (what we did)

In the live demo I ran a classification task: given a short legal text, predict which Eurovoc level-one classes apply. Eurovoc is a multilingual controlled vocabulary maintained by the Publications Office of the EU; level-one classes represent the highest-level thematic categories (e.g., Trade, Environment, Employment, European Union). A sample can be tagged with multiple categories.

We used a subset of the Eurovoc dataset hosted on Hugging Face, sampled across languages, and focused on 21 classes. To demonstrate RFT's sample efficiency, we intentionally used a small training set: 100 training samples and 50 validation samples (150 total). The full dataset contains thousands of samples across many languages. If RFT shows signs of improvement at this scale, you can expand to larger training runs.

Data formats and semantic labels

One concrete lesson from the demo: avoid training on opaque numeric IDs. Eurovoc assigns numeric identifiers to each concept, but identifiers have no semantic meaning for a model. Instead, we translated those IDs into canonical human-readable names (e.g., "Trade", "Agriculture", "Environment"). When the model can reason over meaningful tokens, it generalizes far better. That mapping step is small but impactful.

Balancing the training set

We inspected the category frequency distribution and found it imbalanced. Training on an imbalanced sample can lead the model to “take the easy route” and over-predict frequent classes — a form of reward hacking or shortcut learning. To counteract that risk we applied a balanced sampling strategy: sample uniformly across categories for the training set so the model learns generalizable patterns rather than learning class priors.

For validation, I recommended keeping two views:

Balanced validation — measures generalization across all classes equally.
Production-like validation — measures expected real-world performance under the real label distribution.

Both are useful: the balanced metric answers whether the model learned to handle rare classes, while the production metric tells you real-world impact.

🧾 Designing graders — Turning precision and recall into a continuous reward

A grader is the heart of RFT. It is a programmatic rubric that scores outputs and returns a value the training process optimizes. The grader must be robust, continuous, and aligned with the objective you truly care about.

For classification, the two metrics that matter most are precision and recall:

Precision — of the labels I predicted, how many were correct?
Recall — of the true labels, how many did I find?

Those are often in tension: you can boost precision by predicting fewer labels (and being more conservative), and you can boost recall by predicting more (and risk false positives). To provide a single scalar reward we used the F1 score (a harmonic mean that favors the lower of precision/recall and encourages balanced improvements). F1 yields a continuous score between 0 and 1 that the RFT loop can optimize.

Important grader design advice I stressed:

Return a continuous signal. A binary pass/fail grader produces sparse gradients and incentives the model to guess; a continuous score gives finer feedback and prevents reward hacking.
Be robust to edge cases. The grader must handle malformed or unexpected outputs gracefully and always return a valid score between 0 and 1. Add defensive if-statements to guard parsing and mapping logic.
Make the grader reusable. Save the grader code (stringified or as a function) and reuse it for both evaluation and for RFT. That ensures "you’re comparing apples to apples": the eval used to measure progress is the same metric used to train the model.

"It's critical that evals and RFT use the exact same graders. If you see an increase on the validation set, you want to know it's a valid increase and not an artifact." — Théophile Sautory

✍️ Prompt optimization — Keep it short and consistent

We iterated on prompts and converged on a relatively short prompt that included:

Context describing the task
Clear output schema (we used structured outputs)
List of the 21 classes (canonical names)
Guidelines for decision-making

Two points I emphasized:

Consistency between training and inference. Use the same prompt structure for RFT training and for production inference. If you fine-tune on one prompt and then change the prompt at serving time, you may break the learned behavior. The prompt is part of your environment.
Short prompts are valuable. Short prompts reduce cost, reduce risk of biasing the model excessively with examples, and make training runs faster. If you need to add examples for few-shot style, test whether they actually help — minimal prompts often work best with RFT.

We also used structured outputs (Pydantic-based schema) so the model's outputs are easily parsed. Structured outputs reduce grader parsing errors and make evaluation deterministic. Structured outputs plus validation schema is a powerful combination: you both constrain the model's format and allow rich scoring logic.

🧪 Running evaluations — Evals platform & variance studies

Before we started any training, we ran evaluations to establish baselines and to study variance. Two evaluation patterns we used:

Multiple runs per sample — because rational models like O4 Mini are stochastic, run each evaluation sample multiple times (e.g., 3–9 times) and inspect the distribution of scores. Variance gives you insight into headroom: a sample that sometimes scores 1.0 and sometimes 0.0 indicates there is a reasoning trajectory the model can find; RFT can teach the model to prefer that trajectory.
Variance visualization — plot for each sample: min, mean, max, and variance across runs. A "cross" indicating a max near 1.0 with a low mean is a promising sign: the model can sometimes get it right.

The demo used the OpenAI evals platform to iterate rapidly. The evaluation dashboards help you inspect which prompts, which samples, and which outputs are causing trouble. You can click into individual examples, compare the reference and model output side-by-side, and confirm whether a grader's score makes sense.

"If you see samples where the max score reaches 1.0 but the mean is low, that's headroom for RFT — it tells you the model sometimes finds a correct reasoning path." — Théophile Sautory

🚀 Launching RFT — Job setup, hyperparameters, and checkpoints

Once data, prompt, and graders are ready, we kicked off an RFT job. The platform workflow includes several important steps:

Upload training and validation sets — note that for evals you may upload examples without prompts (so you can test different prompts easily), but for RFT each training sample must include the final prompt + input because the model samples around the prompt environment during training.
Attach the grader and response format — reuse the same grader you used in evals.
Pick the base model — in our demo we used O4 Mini, a reasoning model that benefits from specialization. We chose O4 Mini with "low" reasoning effort for efficiency because the inputs were short and the reasoning task was moderate.
Set hyperparameters — batch size, eval_samples (how many times to run each validation sample per eval), seed, number of steps — these control training stability and costs.
Monitor job outputs and checkpoints — the platform emits checkpoints (the top performing snapshots over time). You can inspect each checkpoint with the linked evals and pick the one that balances metrics and latency for deployment.

Some clarifications about parameters:

Eval_samples affects validation robustness; a higher value reduces noise in validation curves but increases eval time.
Batch size influences variance of training reward; training reward plotted per step can be noisy if batch size is small, since it represents a single sampled batch.
Reasoning effort (low/medium/high) is a knob controlling how verbose chain-of-thought tokens may be. Higher effort can improve results but increases cost and latency.

📈 Interpreting training charts — Reward curve, diagnostics, and behaviors

The training dashboard produced multiple charts that I recommend monitoring closely. Here's what they show and how to interpret them:

Reward curve (training vs validation)

This plot shows reward (e.g., F1) on the y-axis and training steps on the x-axis. The green line often shows batch-level training reward, which is noisy because it reports only the current training batch. The purple dotted line shows the validation reward aggregated across eval samples and is smoother because it's the mean of multiple runs.

Good sign: both curves trend upward. Warning sign: training reward rises while validation is flat — that suggests overfitting or a distribution mismatch between training and validation data.

Precision / Recall / F1 trace

Plotting precision and recall separately for both training and validation sets is important because RFT optimizes a single scalar (e.g., F1), and that scalar hides trade-offs. By tracking both metrics you can pick checkpoints that match your product goals (e.g., favor precision over recall or vice versa).

Example behavior from the demo: at one checkpoint precision was very high but recall relatively low — the model predicted a few very reliable labels. That might be preferred in risk-sensitive scenarios; in others you may prefer higher recall.

Variance study and per-sample distributions

Per-sample plots showing min/mean/max across multiple runs reveal which samples the model can occasionally solve and which are impossible under the current setup. RFT is especially effective when the max is significantly higher than the mean because the model can discover a higher-scoring reasoning path during exploration and then learn to prefer it.

Chain-of-thought / reasoning token trends

The dashboard also displays the number of reasoning tokens the model produces during training steps. Sometimes the model learns a strategy that uses more verbose internal reasoning tokens to reach higher reward. That increases training and inference cost and latency. Monitor this metric because it helps you evaluate whether RFT is encouraging longer chain-of-thoughts and whether that fits your constraints.

Tip: if the model is using more tokens than desired, consider lowering the reasoning effort parameter or adjusting the reward to penalize verbosity.

Grade latency and grader errors

Watch for grader latency and error rates. Graders run inline during training and must be robust (e.g., parsing, response validation). If a grader raises parsing exceptions or frequently times out, that can cause training failures. Build defensive parsing, and ensure the grader always returns a numeric score even on malformed outputs.

🔁 Checkpoints and model selection — How to choose the right snapshot

The RFT job produces multiple checkpoints — typically top-performing snapshots over the course of training. Each checkpoint can exhibit different precision/recall trade-offs, different inference token usage, and different latency characteristics.

My recommended checkpoint selection strategy:

Use eval dashboards to inspect precision, recall, F1, and chain-of-thought tokens for each checkpoint.
Run a small production-like test on each candidate checkpoint using real or representative queries.
Choose the checkpoint that aligns with your product priorities: if safety/compliance is paramount choose higher precision; if coverage is crucial choose higher recall.
Consider latency and token usage: a checkpoint that marginally outperforms another on F1 but doubles inference time may not be worth it.

Remember: checkpoints are real artifacts you can call via the same inference APIs that you use with base models. That makes it straightforward to A/B test in production or behind feature flags.

🧰 Practical checklist — An actionable RFT workflow

Here is a compact checklist I give teams who are getting started with RFT. Treat it like an executable playbook:

Define the task precisely. Is it objective? Is there a canonical "correct" answer? If not objective, consider preference fine-tuning instead.
Baseline with prompting & RAG. Squeeze performance gains from retrieval and prompt engineering first.
Run variance studies on candidate base models. Sample multiple outputs per example. Look for examples where the max >> mean score.
Curate a small high-quality training set. Start with 50–300 examples. Quality > quantity for RFT.
Map opaque IDs to semantic names. Models reason better over meaningful labels.
Balance the training set. Avoid heavy class imbalance in training to prevent shortcut strategies.
Design robust graders. Return continuous scores ∈ [0,1]. Handle parsing errors and malformed outputs.
Keep prompts short and consistent. Reuse the same prompt at training and inference.
Run small RFT jobs and inspect diagnostics. Monitor reward curves, precision/recall trends, token usage, and per-sample variance.
Pick checkpoints and A/B test. Evaluate trade-offs for production deployment (latency, accuracy, risk).
Iterate. Update graders, add curated examples, and expand dataset if needed.

🚧 Common pitfalls and how to avoid them

From experience — and reinforced by the demo and discussion — here are the common failures I see and concrete mitigations:

1. Noise in training data

Problem: RFT is highly sensitive to noisy labels. A few low-quality samples can degrade the learned reward surface.

Fix: Curate the training set aggressively. Remove inconsistent or ambiguous examples. If necessary, relabel or stratify borderline cases.

2. Weak or binary graders

Problem: Binary pass/fail graders encourage guessing and reward-hacking; the model may succeed without developing correct reasoning steps.

Fix: Use continuous, stratified grading functions that reflect degrees of correctness and penalize invalid reasoning or hallucinations.

3. Distribution mismatch between training and production

Problem: Overfitting to a contrived or overly balanced training distribution may reduce production performance.

Fix: Keep a production-like validation set and monitor both balanced and production-weighted metrics. Consider mixing some production-distributed samples into validation.

4. Reward hacking

Problem: The model optimizes a metric in unintended ways (e.g., brevity, verbosity, or format hacks).

Fix: Design graders that penalize pathological behaviors (e.g., hallucinations or format abuse). Add anti-hacking checks to the grader.

5. Uncontrolled chain-of-thought verbosity

Problem: The model learns to generate very long internal reasoning, increasing latency and cost.

Fix: Incorporate token usage into your reward or reduce the "reasoning effort" parameter during training. Inspect chain-of-thought token counts and set constraints.

🏢 Customer spotlight — How Accordance used RFT for tax workflows

I invited David Yu, co-founder and CEO of Accordance, to share how they used RFT in production. I’ll summarize the key takeaways from his experience.

Accordance builds tools for accounting and tax professionals; many of their tasks require precise legal interpretation and policy reasoning. They selected tax strategy optimization tasks for RFT because:

Tax workflows are objectively evaluable. You can compute numeric measures such as tax savings or compliance scores, making them suitable for programmatic grading.
Domain knowledge shifts often (regulatory updates), but the reasoning process around optimization remains stable and valuable to improve.

Practical lessons David highlighted:

Task selection matters. Pick tasks where the model already has a non-zero baseline performance; if your base model scores near zero, RFT may struggle to discover useful trajectories.
Quality over quantity. Accordance sampled a small curated set of high-quality examples rather than a broad noisy corpus. In RFT even a few poor examples can derail learning.
Design multi-part graders. For tax optimization they combined continuous measures (e.g., tax saved) with feasibility and viability checks. This prevented the model from "inventing" unrealistic strategies to game a single numeric objective.
Use industry-standard evals. They validated against an external benchmark (TaxBinge) and observed >40% improvement in relevant metrics after RFT, a result that impacted product value directly.

"Choosing the right task and grader was the most important part. With a small, high-quality training set and a stratified continuous grader, we saw immediate, meaningful improvements." — David Yu, Accordance

💸 Cost, latency, and performance trade-offs

Teams inevitably ask: "How much will this cost? Will latency suffer?" Here are the core trade-offs and pragmatic guidance I offered:

Model selection drives baseline cost. Using O4 Mini and improving it with RFT can let you approach the accuracy of larger models like GPT-4 while staying cheaper and faster. Many teams aim to extract "GPT-4-like" performance from a specialized O4 Mini for production savings.
RFT training itself has cost. There is compute and eval cost to run RFT jobs. Those costs are one-time or infrequent relative to per-query serving costs, so amortize them against expected production traffic.
Inference tokens may increase. If the trained model uses more chain-of-thought tokens, your per-request inference cost and latency may rise. Monitor the chain-of-thought token count and penalize verbosity if needed.
Reasoning effort is a knob. You can trade off reasoning thoroughness and model time by selecting low/medium/high reasoning effort. Low is cheaper and often sufficient for short documents; medium/high is worth experimenting with on complex tasks.

In short: RFT can reduce serving costs by enabling a smaller model to perform at a higher level, but budget for experimentation and watch for increased inference token usage after fine-tuning.

💬 Highlights from the Q&A — Practical answers I gave

During the Q&A several good questions came up. Here are concise answers I shared and why they matter:

Which tasks best suit RFT?

Two conditions make a task ideal for RFT:

It benefits from a reasoning model (i.e., reasoning models noticeably outperform non-reasoning models).
You can construct a formalized, preferably continuous, rubric (grader) to evaluate outputs.

If your task is subjective or style-centric, preference fine-tuning may be more appropriate. If it is factual-knowledge-limited, start with RAG.

Can I use RFT when my data is noisy?

Yes, but cautiously. RFT is sensitive to noise. Clean and curate examples aggressively. Remove conflicting labels and ambiguous cases before training. If necessary use supervised fine-tuning first to standardize formats and then RFT to refine reasoning.

How do I avoid reward hacking?

Design graders that reflect the full goal, not a narrow proxy. Multi-part graders (e.g., measure factual accuracy, feasibility, and conciseness) reduce the incentive to game any single metric. Also, include negative tests to check for hallucination or format abuse.

How many examples should I start with?

Start small: 50–300 high-quality examples. RFT is sample efficient. If you see gains, scale up. If you don’t, re-evaluate graders and prompt design before adding many more examples.

Do I need to use the same prompt at inference?

Yes — keep the prompt consistent between training and inference. The prompt is part of the task environment that RFT optimizes against; changing it later risks degrading performance.

📚 Additional resources and next steps

To help teams reproduce what we did, here are the key artifacts and resources I referenced:

OpenAI Build Hours code repo: https://github.com/openai/build-hours — a practical companion with notebooks and scripts.
RFT Cookbook: https://cookbook.openai.com/examples/reinforcement_fine_tuning — patterns, grader templates, and design guidance.
RFT Use Case Guide: https://platform.openai.com/docs/guides/rft-use-cases — how to decide when to apply RFT.
Sign up for upcoming Build Hours and workshops to watch live demos and ask questions: https://webinar.openai.com/buildhours

I also encourage teams to instrument their experiments thoroughly: save graders, keep versioned prompts, store training and validation splits, and preserve checkpoints. Reproducibility saves time and helps you iterate faster.

🧩 A concrete example — Minimal code and grader pseudocode

To make the discussion more tangible, here's the minimal conceptual flow of the RFT loop and an example grader pseudocode (conceptual, not runnable):

Prepare training set: each item contains id, prompt (including context), and reference labels (semantic names).
Define structured response schema (e.g., JSON array of labels or Pydantic model).
Implement graders:

Example Pseudocode Grader (classification F1):
def f1_grader(reference_labels, predicted_labels):

# convert to sets, guard parse errors

if parse_error: return 0.0

precision = intersection / len(predicted) if predicted else 0

recall = intersection / len(reference) if reference else 0

if precision + recall == 0: return 0.0

f1 = 2 * (precision * recall) / (precision + recall)

return clamp(f1, 0.0, 1.0)

A few notes on robustness:

Always clamp scores to [0, 1].
Guard parsing steps (structured outputs fail gracefully).
Log grader errors for quick debugging.

Design graders with multiple components if your task requires nuance (e.g., a numeric score combined with a feasibility boolean). The final reward can be a weighted sum or more complex composition, but simplicity often works best early on.

🔚 Conclusion — When to reach for RFT and how to succeed

Reinforcement fine-tuning is a powerful and pragmatic technique when the goal is to improve how a model reasons rather than to add knowledge. It is especially useful when:

Your task is objective or can be measured with an agreed rubric.
Your base reasoning model has non-zero baseline performance.
You can engineer a grader that returns a continuous signal and penalizes undesirable behavior.
You have the capacity to curate a small set of high-quality training examples and to iterate.

My closing advice: treat RFT like crafting an environment for the model. The grader, prompt, and training dataset define that environment. Invest time in designing a clean, robust grader and a high-quality sample set before scaling up. Monitor per-sample variance and per-checkpoint trade-offs, and select the checkpoint that meets your product's accuracy and latency requirements.

If you follow that playbook, RFT can transform a general reasoning model into a domain expert, often with far fewer labeled data than you might expect.

📢 Final notes and invitations

If you're building or planning an RFT experiment, I encourage you to start small: pick 50–200 high-quality examples, create a continuous grader, run variance studies, and iterate. Share results with your team and consider external benchmarks where available. For folks who want to replicate our demo, check the OpenAI Build Hours repo and the RFT cookbook linked above.

We continue to run Build Hours with practical demos — if you want live walkthroughs, sign up for upcoming sessions. And if you have questions about grader design, dataset curation, or checkpoint selection, I welcome you to reach out through developer channels; these problems are highly solvable with the right instrumentation and experimental discipline.

Thanks for reading — I hope this article helps you design stronger, more reliable reasoning agents with reinforcement fine-tuning.