Salesforce study finds 42% compute savings by distilling RL-trained experts

Training one AI model to be good at everything is expensive. Salesforce AI Research says there may be a cheaper route: train several specialists, then pour what they know into a single model.

In a preprint released on November 5, 2026, the team reports that this approach matched the performance of joint reinforcement learning on a combined math-and-code setup while using 42% fewer total training FLOPs.

What the researchers actually tested

The paper is titled “Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO” and is posted as arXiv:2611.08453.

GRPO stands for Group Relative Policy Optimization. It is a reinforcement learning technique that trains a model using verifiable outcome rewards, meaning the model gets credit when its result can be checked as correct.

OPD stands for on-policy distillation. The study pits these two approaches against each other on a simple scoreboard: how much held-out accuracy does each method deliver for every unit of training compute spent?

To answer that, the researchers measured accuracy on held-out tasks against training FLOPs across multiple domains, including math and code, at different task difficulties and with different teacher setups.

The headline result: specialists, then a merge

The standout finding involves what the paper calls multi-teacher OPD. Instead of training one model on math and code at the same time with GRPO, the researchers first trained separate GRPO “expert” checkpoints for each domain.

They then distilled those experts into a single student model. In the math-plus-code setting, that student reached performance comparable to joint GRPO training, with a total compute bill 42% lower.

The merge step itself turned out to be cheap. According to the research, distillation accounted for just 5.2% of the cost of training the specialists in the two-domain case.

When the setup expanded to four domains, distillation’s share rose to 8.0% of specialist training FLOPs.

Where distillation fell short

The paper does not crown distillation as a universal winner. When OPD relied on external post-trained teachers, meaning models trained elsewhere rather than in-house experts, it generally underperformed GRPO on a per-FLOP basis.

Those external teachers also tended to make student responses longer than necessary.

There was a bright spot on the teacher front, though. Self-GRPO teachers, built from the student’s own earlier checkpoints, proved highly efficient at recovering single-domain accuracy. The research summary describes reductions in distillation FLOPs of up to a factor of 50 in that setting.

Why the result matters for AI training budgets

The study fits a broader effort at Salesforce to make large language model post-training cheaper. It also builds on ongoing development of pipelines that use reinforcement learning with verifiable rewards, a family that includes GRPO.

The four-domain numbers hint at scalability. Distillation’s share of cost rose from 5.2% to 8.0% as domains doubled, which suggests the merge step stays relatively affordable as the expert lineup grows, at least within the configurations tested.

There are caveats worth keeping in view. This is a preprint, so it has not yet been through peer review, and the headline 42% figure comes from a specific two-domain math-and-code comparison.