Skip to content
Train

Measure a model, then make it better.

Start with a baseline on any environment, then train with reinforcement learning against its verifier: the model practises, gets scored, and learns from what scored well. Each round is kept only if it beats the best score so far.

Start a project Read the docs
best 0.7101Baselinemeasured first02Each roundscored on held-out tasks03Reverted if worsenever kept04Best keptthe one you ship
Two goals

Evaluate, or improve.

GSM8K200 questions · seed 0

Qwen2.5-1.5B14.0%
SmolLM3-3B9.5%
Qwen2.5-0.5B5.0%
Run #4 errorednot scored
The pipeline

Baseline, check, train, keep the best.

Every project runs the same pipeline, so a better number always means a better model.

1

Baseline

Measure the model before anything changes, on a fixed number of episodes.

2

Check

Confirm the environment's tier, so the reward really separates good from bad.

3

Train

Run rounds of the learner you picked against the environment's verifier.

4

Keep the best

Held-out tasks after every update; a worse round is reverted.

Held-out gate

Training that can't go backwards.

train() holds out tasks and evaluates after every update. An update that makes held-out performance worse is reverted, so the best model so far is the one you keep.

  • Held-out tasksScored after every update, never trained on.
  • Reverted if worseA round that lowers the held-out score is rolled back.
  • Learning signal reportedYou see what fraction of groups carried any signal at all.

Each round kept only if it improvesGRPO · held-out

r1
r2
r3
r4
r5
r6

Round 4 was worse, so it was reverted0.71

Learners and trainers

Built in, or the one you use.

Ready means run end to end for real. Beta means the job is generated and tested and the reward path checked against real environments, but not yet run end to end on a GPU.

StatusWhat it runs
Nenyax learnersReadyIn-context, GRPO, DAPO, RLOO, RFT, DPO
TRLBetaGRPOTrainer or DPOTrainer
UnslothBetaGRPO with 4-bit loading and LoRA
verlBetaGRPO, PPO, RLOO, REINFORCE++
prime-rlBetaOn a Verifiers environment wrapping Nenyax
OpenRLHFBetaPPO with a remote reward server
TinkerBetaHosted LoRA, environment stays local
Around every run

Everything a run keeps, and where it runs.

Settings, data and machines stay yours; the result stays repeatable.

Held-out reward0.81

learner
grpo
model
Qwen2.5-0.5B-Instruct
lr
3e-6
rounds
10
group_size
8
batch
4
max_new_tokens
128

examples/configs/grpo_local.toml

A run

Every setting next to its curve

The learner, model, learning rate and rounds are kept with the result, so a run can be repeated.

Your machine runner online

OPENAI_API_KEY HF_TOKEN WANDB_API_KEY
Outbound only:heartbeatclaimeventsfinish
On your machines

Keys stay home

Training runs on compute you control; Nenyax owns none.

Langfuse LangSmith Phoenix Claude Code sessions
scrubbed→
Environment tasksSFT pairsDPO preferencesDistillation set
From your traces

Datasets to start from

SFT pairs, DPO preferences or a distillation set, made from your own traces.

StackChange

ModelvLLM · Qwen3-8B
ComputeRunPod · your account
SandboxModal
Trainerverl
Your stack

Swap the trainer, keep the environment

Move from the built-in GRPO to verl or TRL without touching the environment or its verifier.

From your terminal

Baseline, then train.

zsh
nenyax project create my-eval --env reasoning-gym/basic_arithmetic --model anthropic/claude-sonnet-5-5 --runners
nenyax project run my-eval baseline --watch
Questions

Good to know.

LLMs scored by a verifier, agents doing tool use and long tasks in a sandbox, and RL policies for control and games. Robots are coming.

Put a model through its first round.

Measure a baseline, then improve from there on your own stack.

Start a project Read the docs
nenyax train reasoning_gym:basic_arithmetic \
  --learner grpo --provider vllm
kept only if it improves