AReaL 2.1 keeps RL services separate while colocating actor and rollout

AReaL project head Yi Wu (left) with Stuart Russell, his former UC Berkeley mentor. Photograph supplied by Wu Yi to 36Kr and published by KrASIA.[7]

At 10:23 UTC on August 25, the AReaL project published version 2.1.0. Its most consequential change looks like a reversal: after version 2.0 split reinforcement learning into independent training, inference, agent, and weight-update services, 2.1 can place the actor and rollout work back on the same GPUs through AWEX.[1][2]

It is not a retreat from modularity. It is a distinction between logical separation and physical placement. Training and inference retain different roles, interfaces, and weight versions; an operator can now colocate them when alternating GPU demand makes shared hardware attractive, or keep them separated when scale and failure isolation matter more. Version 2.1 also adds grouped colocation to AReaL's HTTP-based Ray scheduler, native AWEX colocation for Qwen3-VL, and a delta-weight transfer path for separated AdamW training.[1]

That placement story is only half the release. A dense run of fixes now attributes output tokens to the policy version that actually served them, drops retry-orphan completions that could split a trajectory, preserves singleton online rewards, rejects incomplete sampling evidence, pauses proxy workers during weight updates, and protects model registration with an admin key.[1] These lines read like maintenance. In online RL, they define whether a gradient is attached to trustworthy experience.

The honest release note is therefore sharper than a speed claim. AReaL 2.1 makes its July service architecture more deployable and its learning record more internally consistent. It does not publish an end-to-end 2.1 benchmark showing that colocation is faster, cheaper, or equally stable on a named workload. Nor does it complete the governed “self-evolving agent” loop described by the project's own research agenda.[1][4]

Version 2.0 separated the jobs; 2.1 makes placement a choice

The original AReaL system attacked idle GPUs in large-model reinforcement learning. Instead of making rollout generation wait for training, or training wait for the longest generated answer, it ran the two asynchronously. That created its own correctness problem: a training batch could contain tokens sampled under several older policies. AReaL tracked policy versions, bounded data staleness, and changed the PPO objective to account for the behavior policy that produced each sample.[6]

In the authors' controlled math-and-code experiments, the system reported up to 2.77× the training speed of synchronous systems using the same number of GPUs, with matched or improved final performance. Those numbers belong to the earlier AReaL evaluation setup; they are not measurements of the new 2.1 placement modes.[6]

Version 2.0, released July 1, moved that asynchronous engine behind four service boundaries: training, inference, agent, and weight update. It added a shared command-line surface and examples that connect Hermes and software-engineering agents to the training loop.[2] An agent could keep its planner, tools, memory, and sandbox while redirecting ordinary model calls through an AReaL-managed endpoint. The service would capture token-level interaction data and feed completed, rewarded sessions into online policy training.[4][5]

That decomposition improves replaceability, but it does not answer where every process should run. Separate training and rollout clusters can each be sized for their own workload. They can also strand memory and compute when one phase is busy and the other is quiet. Colocation tries to reclaim that slack without fusing the software back into one opaque process.

AWEX is the hinge between one GPU pool and two model layouts

Training and inference do not necessarily store a model the same way. The actor may use one parallel layout and optimizer state; the rollout engine may shard weights differently for fast generation. Moving a new policy between them is therefore more than copying a checkpoint.

AWEX, an InclusionAI weight-exchange project used by AReaL, separates that handoff into a writer on the training side, a reader on the inference side, and a metadata service that lets both construct a deterministic transfer plan. Its published design supports both colocated and separated deployments, converts between training and inference layouts, and transfers only the shards a receiving worker needs.[3]

AReaL 2.1 turns that capability into several concrete operating paths. The release adds AWEX-based actor-rollout colocation, grouped colocation under Ray, Qwen3-VL support on the native colocated path, and AdamW delta transfer when training and inference remain separated. It also moves the AWEX dependency to version 0.8.1.[1] The pattern is coherent: preserve the service contract, then let the scheduler choose whether the services share a machine, a GPU group, or only a weight-transfer channel.

What is missing is just as important. The AReaL release note provides no paired table comparing separated and colocated runs on wall-clock training time, peak memory, weight-sync latency, trajectory staleness, or final task reward.[1] AWEX's own repository reports fast synchronization at much larger scales, but that is a framework-level claim, not a measured AReaL 2.1 result.[3] Until an apples-to-apples run appears, colocation is an available placement strategy—not a proven universal win.

The quiet fixes protect the learning record

Online RL depends on a stricter evidence chain than ordinary model serving. A response has to be tied to the exact policy that generated it. Its sampled tokens and log probabilities must be complete. The reward must belong to the right session or group. A retried request must not become two contradictory training examples. And no worker should continue serving halfway through a weight transition.

Version 2.1 touches every link in that chain.[1]

The serving-version fix matters because AReaL's asynchronous objective compares the current policy with the behavior policy behind a trajectory. If output tokens are attributed to the controller's convenient version rather than the version resident on the serving worker, the importance ratio is built on the wrong denominator. The new incomplete-evidence rejection follows the same logic: dropping a malformed sample is safer than treating missing sampling metadata as valid experience.[1][6]

Retry cleanup addresses a more familiar distributed-systems failure. An agent SDK can time out after the server generated a completion, retry the same prompt, and consume only one of the two answers. If the orphan is exported independently—or if the two completions split what should be one session—the trainer sees a history the user never actually experienced. AReaL 2.1 now drops those retry orphans and forwards that cleanup through the version-2 path.[1]

Reward preservation is equally substantive. The release fixes a case where a singleton online reward could be lost, and it adds grouped reward-normalization controls plus rejection-aware token and log-probability statistics.[1] None of those features can tell whether a reward represents genuine success. They can ensure that, once an operator defines the signal, the system does not silently move, erase, or normalize it under the wrong grouping rule.

The security fix belongs in the same category of maturity. Requiring an admin key for /register_model closes a server-side request-forgery path at a control endpoint.[1] A model registry that can be steered by an unauthenticated caller is not merely an API flaw; it can redirect the compute and data plane that the learning loop trusts.

A cleaner policy-update loop is still only one kind of evolution

The AReaL 2.0 technical report draws a useful boundary around the word “self-evolving.” It proposes three larger components: a standard trajectory protocol carrying observations, actions, delayed rewards, versions, provenance, and training eligibility; an enterprise data proxy that redacts and isolates records while supporting replay; and a control plane that chooses whether a failure calls for a memory insertion, skill patch, prompt or tool-schema edit, model update, rollback, or no action.[4]

The authors explicitly describe AReaL 2.0 as a prototype of one branch: online policy-model weight updates. Full trajectory governance, counterfactual replay, tenant-aware privacy, and automatic selection among several intervention surfaces remain outside that implementation.[4] Version 2.1 hardens the branch. It does not erase the boundary.

That distinction prevents a category error. Deterministic sampling can make an interaction reproducible; it cannot make the underlying data eligible for training. Correct policy attribution can make an update mathematically coherent; it cannot show that the update should pass a safety regression. An authenticated model-registration endpoint can protect the control plane from one attack; it cannot decide which team's traces may change a shared model.

The next proof is an end-to-end placement benchmark

A convincing 2.1 evaluation would compare separated and colocated actor-rollout runs on the same model, workload, hardware budget, and reward setup. It would report useful-work throughput, peak GPU memory, offload and weight-sync time, rejected or stale trajectories, restart recovery, and final held-out task quality. For online mode, it should also inject retries and delayed rewards, then show that the exported trajectory is the one the agent actually experienced.

The falsifier is straightforward. If shared placement merely exchanges idle GPUs for offload stalls, memory pressure, and longer weight transitions, AReaL's logical-separation story will survive but the 2.1 colocation thesis will not. If the same service graph can move between separated and colocated hardware while preserving policy attribution, reward integrity, and final learning quality, the release will have demonstrated something more durable than a speed trick: an RL system whose software boundaries no longer dictate its physical topology.

For China's AI stack, that is the signal worth watching. The frontier contest is not only about releasing another model. AReaL 2.1 shows Ant-, HKUST-, and Tsinghua-affiliated systems work moving into the less glamorous layer where live agent experience becomes—or fails to become—reliable training data.[4]

cronfeed.work