What we learn by running everything.
Notes from wiring every RL format into one contract and measuring what each environment actually does, written down with the evidence.
What we learned building Nenyax v0.1
Every format fits one of three modes. Across six formats there were no exceptions: an environment is stepped from outside, runs its own evaluation loop, or is a task graded by tests. So mode became a required field, and one call, rollout(), drives all of them.
Plug-and-play is a spectrum, so we measure it. The same driver reaches different tiers on different environments: public OpenEnv Spaces are evaluable, seeded servers trainable, and reasoning-gym audited. Reproducibility is rarely declared, so an environment is only resettable when seeding is asserted.
Honest numbers need honest failure. train() once reported a clean 0.000 while every episode was erroring on a wrong port; it now raises with the first error. And low scores from a small model were genuine: it answered without the required tags, which is exactly what training on these environments teaches.
At a glance
Gymnasium 1.3, reasoning-gym 0.1.25, OpenEnv 0.6, NeMo Gym 0.6, Verifiers 0.3.1 and Harbor 0.23, plus one real model, Qwen2.5-1.5B served by mlx-lm.
Read the full notesWhat we measureddocs/learnings.md
Results by format.
Which tier each format reached when driven end to end, and why.
| Mode | Tier reached | |
|---|---|---|
| reasoning-gymthe gold answer gives a free oracle | step | audited |
| NeMo Gyma clean /verify contract over HTTP | step | audited |
| Verifiersgsm8k scored gold 1, wrong 0 | rollout | audited |
| Harbororacle and nop give a built-in audit | task | audited |
| Gymnasiumno oracle exists, so no audit | step | trainable |
| OpenEnvseeded servers; public Spaces are evaluable | step | trainable |
Four threads we keep pulling.
nenyax checkreasoning_gym:basic_arithmetic
nenyax checkreasoning_gym:basic_arithmetic
Where the time goes.
Per-episode cost varies by four orders of magnitude across formats. Scripted reasoning-gym episodes take under a millisecond; a Harbor agent in a container takes 16 to 25 seconds, dominated by container and agent setup.
That is why pooling Verifiers serving and Harbor containers across episodes is near the top of what we improve next, along with Docker throughput, which sat at 1.3 episodes per second under load on Docker Desktop.
Timingsper episode
Good to know.
Run the experiments yourself.
Everything here comes from runs you can repeat with the SDK.