Does RL actually teach LLMs to reason, or just to reason within its training distribution?

Introduction

Reinforcement learning has become the default way to push language models past what supervised fine-tuning can do on math and reasoning benchmarks. Train a model with Group Relative Policy Optimization (GRPO) on a pile of math problems, and accuracy climbs. The harder question, and the one that actually matters for anyone deploying these models, is what that improvement is made of. Does the model learn to reason, in some portable sense, or does it learn a strategy that only works inside the box it was trained in?

We ran a controlled experiment to find out; training three variants of the same model on the same data with GRPO, then testing how far the resulting skills actually travel.

The setup

We took Qwen3-4B-Thinking and trained it with GRPO on the MATH dataset under three conditions. For each prompt $q$, GRPO samples a group of $G$ responses $\{o_1, \dots, o_G\}$, scores them, and computes each response's advantage relative to its own group:

Because the group mean serves as the baseline, GRPO needs no separate value network. The policy is then updated with the usual clipped surrogate,

with $G = 4$ and no KL penalty applied during training ($\beta = 0$).

The three conditions differ only in $r_i$:

• GRPO-MATH: standard GRPO, no tools, binary correctness: $r(o_i, q) = +1$ if the final answer matches ground truth, $-1$ otherwise.
• GRPO-MATH-TIR: the same reward, but the model can call a Python interpreter mid-reasoning (tool-integrated reasoning); it is still scored on final-answer correctness only.
• GRPO-MATH-TIR+PR: tool use, plus a process reward of $\alpha = 0.4$ for every successful tool call:

where $K_i$ is the number of tool calls in response $o_i$ and a call counts as successful if it returns non-empty output with no Python error.

As the results below show, this extra term is enough to change how often the model calls tools.

Training ran on the Leonardo HPC cluster (16×A100s), using VERL and VerlTool. Rather than stop at "does accuracy go up," we split evaluation into three separate questions:

1. Does it generalize across prompts? Same math, different phrasing: paraphrased problems, and a Bengali translation.
2. Does it generalize across tasks? Structurally different reasoning problems: Game of 24 (combinatorial search) and ZebraLogic (constraint satisfaction), plus AIME 2026 as a harder in-domain math benchmark.
3. Does it generalize across environments? What happens when a model trained to lean on a tool is evaluated in a setting without needing one?

What we found

Within the training domain, RL delivers, and keeps delivering. MATH accuracy rose from a 67.16% baseline to 83.26% (GRPO-MATH), 90.02% (GRPO-MATH-TIR), and a peak of 90.62% (GRPO-MATH-TIR+PR). The two tool-using conditions level off at around 90% from about epoch 35, while GRPO-MATH was still climbing at epoch 40. Tool use raises the ceiling by roughly 7 points over pure chain-of-thought; process rewards add only a marginal gain on top of that.

The gains carry over to AIME, which the model never trained on. GRPO only ever saw problems from the MATH training set, and no AIME problems were used for training under any condition. Even so, AIME 2026 pass@8 roughly doubled, from 33.33% for the base model to a peak of 66.67% under all three conditions. AIME 2026 also took place after Qwen3-4B-Thinking was released, so these problems cannot have appeared in its pretraining data either. Because AIME problems are harder than MATH and the model never saw them, this is the clearest sign that RL on MATH improves mathematical reasoning that carries over to new problems, rather than fitting the training set.

The effect of process rewards is clearest early: by epoch 7, TIR+PR is already at 83.66%, 2.5 points ahead of TIR (81.16%) at the same epoch. The $+0.4$ per successful tool call provides a denser learning signal in the epochs where the model has not yet learned to reliably produce correct final answers. It also changes the habit the model forms. Over the first 11 epochs, tool use intensity rises from 1.20 to 1.375 under TIR+PR, while under plain TIR it falls from 1.23 to 1.16: every successful call earns $\alpha$ regardless of whether it was necessary, so where TIR learns to use tools selectively, TIR+PR learns to use them habitually.

Prompt-level robustness is genuinely strong. Accuracy on paraphrased problems stayed within 2 points of the original English problems at every epoch, for all three conditions. The Bengali translation behaved differently over time. After the first epoch, Bengali was up to 8 points ahead of English, plausibly because Bengali math phrasing carries less of the idiomatic, stylistically varied baggage that English problem sets tend to have, so the model has less surface noise to see past. As training went on, English caught up and overtook it (around epoch 5 for TIR+PR, epoch 12 for TIR and epoch 30 for GRPO-MATH), and by epoch 40 Bengali was at most 3 points behind. Whatever RL is doing here, it isn't just memorizing prompt templates.

Cross-task generalization is where it breaks down, and it breaks down asymmetrically. On Game of 24 (draw four numbers, hit 24 using + − × ÷), the tool-trained models don't just fail to transfer, they fall below the untrained baseline. From epoch 3 to epoch 11, GRPO-MATH-TIR scores only 5–7% against a 24.89% baseline, while reaching 84% on MATH, and by epoch 30 it has recovered to only about 12%. GRPO-MATH, which never learned to lean on a Python interpreter, generalizes far more gracefully, climbing to 45.67% by epoch 29. The same pattern shows up on ZebraLogic: GRPO-MATH stays roughly flat at baseline (it never picks up transferable logical reasoning, but it doesn't actively lose ground either), while the TIR conditions crash from 72.4% to around 40% after a single epoch and only partially recover, reaching about 59–62% by epoch 30.

The mechanism is intuitive once you see it: MATH problems are almost always solvable with a single, direct computation. The model learns "when in doubt, write code that computes the answer." That strategy is actively wrong for Game of 24, which needs combinatorial search over many candidate expressions, and for ZebraLogic, which needs constraint propagation, not arithmetic. Tool-integrated training doesn't just fail to transfer, it teaches a habit that gets in the way on structurally different problems. Process rewards push the model to call tools more often, but the OOD results don't show that this clearly makes things worse. On Game of 24, TIR+PR drops lower early on (3.67% at epoch 5, against 6.02% for TIR) but recovers further (16.08% against 11.82% at epoch 30). On ZebraLogic, it peaks about 3 points lower than TIR (58.9% against 61.6%).

Skills transfer, but they cost you the source domain. We also asked: if you take a MATH-TIR checkpoint and fine-tune it on Game of 24, does prior tool-use experience help? Yes, game24-transfer reaches 82.82% accuracy, about 6 points above training from scratch, and gets there faster. But MATH accuracy erodes the entire time, falling from 89.84% to 82.30% over 42 epochs of Game of 24 training, a textbook case of catastrophic forgetting. The model trained from scratch drops about 5 MATH points in the first epoch (67.16% to 62.06%) and then stays flat at around 60%; the transfer model steadily trades away what it knew. Practically, this argues for early stopping, around epoch 20–25, Game of 24 performance has mostly converged while MATH degradation is still moderate.

The takeaway

RL post-training on LLMs doesn't produce one thing called "reasoning" that either transfers or doesn't. It produces a strategy, tuned tightly to the computational assumptions of the training task that generalizes well across surface variation (phrasing, language) and poorly across structural variation (a different kind of problem that needs a different kind of solving). Tool integration is a clear performance win in-domain, but it adds a specific, measurable liability: the model doesn't just learn to use tools, it learns a tool-use pattern shaped by the training distribution, and that pattern actively interferes when the new task doesn't fit the same mold.

For anyone deploying RL-trained reasoning models, the practical implication is to stop treating "reasoning improved" as a single number. It's worth asking, separately: improved at what, robust to what kind of variation, and built on what assumptions about the environment it'll be running in.