RL with Verifiable Rewards (RLVR)

RLVR trains a model with reinforcement learning on tasks rather than demonstrations. The model attempts each task in a sandboxed environment, a programmatic checker verifies the outcome, and the reward comes only from what the checker can prove — never from the model's own text or an LLM judge.

ToothFairyAI implements RLVR as agentic GRPO: each training rollout is an episode where the model works on the task using terminal tools (run_command, read_file, write_file, list_dir) inside an isolated per-rollout sandbox, and the group-relative advantage is computed from the checker's verdict across numGenerations attempts per task.

When to use RLVR

Method	Use when	Data you need
SFT (`conversationalTraining`)	You have good demonstrations to imitate	100s–1000s of instruction–answer pairs
DPO / KTO	You have preference signal (better/worse answers)	Preferred vs non-preferred (or binary labelled) responses
GRPO (standard)	You have prompts + a reward heuristic	Prompts, reward functions
RLVR (GRPO + `rl.enabled`)	Success is machine-checkable (files created, tests pass, commands exit 0)	Task definitions with verifiers

RLVR shines for agentic / coding / formatting tasks with objective success criteria. It does not suit open-ended quality judgments — if a human would have to score the answer, use DPO/KTO instead.

How it works

flowchart LR
    A[rlTask document<br/>prompt + checker] --> B[Task manifest JSONL]
    B --> C[GRPO training loop]
    C --> D[Rollout: model + tools<br/>in sandboxed env]
    D --> E[Checker runs against<br/>final sandbox state]
    E -->|verifiable reward 0..1| C
    C --> F[LoRA adapter]

Author tasks as rlTask documents in the Knowledge Hub (see below).
Start a job with training_type: grpoTraining and rlvr: true — the dataset step builds a task manifest (one task_spec per line) instead of a conversation dataset.
Training: for each task, the model attempts the task numGenerations times (default 8) in isolated sandboxes. The checker scores each attempt; GRPO pushes probability mass toward the successful strategies relative to the group.
Outcome: a LoRA adapter, downloadable and versionable like any other fine-tune.

The rlTask document

An rlTask document carries a task_spec:

{
  "prompt": "Create a Python file named add.py that prints the sum of 2 and 3",
  "env": "tfcode",
  "max_steps": 8,
  "setup_files": { "README.md": "optional pre-seeded files" },
  "setup_commands": ["optional setup shell commands"],
  "checker": {
    "type": "exit_code",
    "config": { "command": "python3 add.py | grep -q 5", "expected_code": 0 }
  }
}

Field	Description
`prompt`	The instruction shown to the agent (required)
`env`	Rollout environment — `tfcode` (default, the only one in v1)
`max_steps`	Tool-call budget per episode (1–64, default 16)
`setup_files` / `setup_commands`	Optional sandbox pre-seeding (files written, then commands run, before the episode)
`checker.type`	One of the four verifiers below
`checker.config`	Type-specific configuration

The four checkers

Checker	Config	Reward
`exit_code`	`{command, expected_code?}`	1.0 if the command exits with `expected_code` (default 0), else 0.0
`file_exists`	`{paths: [...]}`	Fraction of required paths present
`pytest`	`{test_paths: [...]}`	Fraction of passing tests
`script`	`{script}`	Your Python script prints a float in [0,1] (shaped reward) or exits 0/1

Reward design

Binary (0/1) checkers are the most hack-resistant. Shaped rewards are safe only when they are themselves programmatic (e.g. fraction of tests passing). Never reward the model's self-reported result — always verify from the sandbox state.

Creating tasks via the API

curl -X POST https://api.<region>.toothfairyai.com/doc/create \
  -H "x-api-key: $KEY" -H "Content-Type: application/json" \
  -d '{
    "workspaceid": "$WS",
    "data": [{
      "autoParse": true,
      "rawTrainingData": {"task_spec": {
        "prompt": "Create greeting.txt containing the word Bonjour",
        "env": "tfcode",
        "max_steps": 8,
        "checker": {"type": "exit_code",
                    "config": {"command": "grep -q Bonjour greeting.txt"}}
      }},
      "type": "rlTask",
      "userid": "$USER_ID",
      "title": "Greeting task",
      "topics": [],
      "status": "published",
      "scope": "training"
    }]
  }'

In the app, the same document is authored visually with the RL Task and RL Checker blocks in the editor (single instance of each per document, like the Tools Definition block).

The `rl` configuration

Pass the RL sub-config on /start:

POST /finetuning/start/{id}
{
  "config": {
    "rl": {
      "enabled": true,
      "numGenerations": 8,
      "maxCompletionLength": 1024,
      "maxToolCallingIterations": 16,
      "lossType": "dapo",
      "scaleRewards": "batch",
      "entropyCoef": 0.05,
      "generationEngine": "auto"
    }
  }
}

Key	Default (clamped)	Meaning
`enabled`	`false`	Activates the RLVR path (only with `grpoTraining`)
`env`	`tfcode`	v1 supports the tfcode terminal sandbox only
`numGenerations`	8 (4–16)	GRPO group size G — more generations = better advantage estimates at higher cost
`maxCompletionLength`	1024 (256–4096)	Max completion tokens per generation
`maxToolCallingIterations`	16 (4–32)	Tool-call turns per episode
`lossType`	`dapo`	`dapo` (recommended), `dr_grpo`, or `sapo`
`scaleRewards`	`batch`	`batch` (robust default), `group`, or `none`
`entropyCoef`	0.05 (0–0.5)	Entropy bonus against policy collapse
`generationEngine`	`auto`	`vllm` \| `continuous_batching` \| `plain` \| `auto` (see below)
`maxSteps`	0 (derive)	Cap optimizer steps
`pipPackages`	`[]`	Extra pip packages installed into the training container at startup (see below)

Additional platform-enforced behaviour for RLVR runs: the learning rate is capped at 2e-6 (RL needs small updates), the KL anchor beta is forced to 0.0 (DAPO/Dr.GRPO consensus — also saves a reference model's worth of GPU memory), and spot training is disabled (checkpoint-resume with colocated generation engines makes spot churn costly).

Extra packages (`pipPackages`)

The RL sandbox is directory-isolated, not interpreter-isolated: rollout commands and checkers run the training container's Python interpreter. pipPackages lets you install additional libraries into that environment once at job startup — every rollout and every checker can then import them:

"rl": { "enabled": true, "pipPackages": ["pandas>=2.0", "requests"] }

PEP 508 specifiers only (PyPI): name, name[extras], name==1.0, name>=2.0,<3.0. URLs, git+ references, local paths and pip CLI flags are rejected with a 400.
Max 50 entries; the install runs before model loading and the job fails fast if any package can't install (a half-installed environment would corrupt rewards).
Careful with pins: a package that conflicts with the training stack (trl/transformers/unsloth) can break the run — pin conservatively and prefer pure-Python additions.

Model eligibility

RLVR requires models with prefix-preserving tool-calling chat templates. The GET /finetuning/models endpoint exposes a supportsRl flag; the eligible set is currently:

unsloth/Qwen3-0.6B / 1.7B / 4B-Instruct-2507 / 8B
unsloth/Llama-3.2-1B / 3B-Instruct
unsloth/gemma-3-1b-it

We recommend starting at 0.6B–1.7B — RL runs are 10–50× longer than SFT runs and small models iterate fastest.

Generation engines

Engine	When	Notes
`plain`	Default on fp16-only GPUs (e.g. T4 `ml.g4dn`)	transformers `generate()` — most compatible, slowest
`vllm`	bf16-capable GPUs when `trl[vllm]` is available	Colocated vLLM — fastest generation, weight-sync to the trainer each step
`continuous_batching`	Explicit opt-in	transformers continuous batching — middle ground
`auto`	Default	Picks `plain` on fp16 GPUs, `vllm` elsewhere

Monitoring an RL run

The status endpoint's metrics include RL-specific series:

rewards/TfcodeEnv/mean — the number that matters: task success rate. It should trend up.
rewards/TfcodeEnv/std — within-group reward spread; healthy RL keeps this > 0 (all-same rewards = no learning signal for those tasks).
frac_reward_zero_std — fraction of tasks where every rollout got the same reward (all-solved or all-failed). High values mean the task set is too easy/hard for the model — re-tier it.
entropy — policy diversity; a collapse toward 0 means the entropy bonus (entropyCoef) should be raised.
completions/mean_length + completions/clipped_ratio — length blowup is the classic reward-hacking symptom.

Pros, cons and caveats

Pros

No demonstrations needed — only tasks + verifiers (cheap to author).
Directly optimises task success, not imitation of demonstrations.
Extremely effective for tool-use/agentic behaviour (the model practices using tools in a real sandbox).
Rewards are objective and reproducible — no judge drift, no annotation disagreements.

Cons

Slow and compute-hungry: every step generates G full episodes with tool calls. Budget 10–50× an SFT run.
Only as good as your checkers: a checker that can be gamed will be gamed (e.g. file_exists on a path the agent can write garbage to — prefer checkers that verify content, like exit_code with grep, or pytest).
Task difficulty matters: if the model solves all tasks (or none), frac_reward_zero_std → 1 and learning stalls. Curate a task pool where the base model succeeds on ~10–90% of tasks.
Not for subjective quality: no verifier, no RLVR.

Caveats specific to ToothFairyAI v1

One environment (tfcode); custom RL containers are on the roadmap (the rl.env field reserves the slot).
The sandbox is an isolated directory per rollout with subprocess-isolated command execution, timeouts and output caps — suited to file/script manipulation tasks, not network-dependent tasks.
num_generations must divide the effective batch (batchSize × gradientAccumulationSteps); the API clamps values for you.
An SFT cold-start before RLVR is recommended when the base model can't follow the tool-calling format at all (see the recipe note below).

Recipe: SFT then RLVR

For very small models (≤1B) that have never used tools, a short SFT stage first (500–2k tool-call demonstrations) stabilises the RL phase — the model must already produce the tool-call format before GRPO can reinforce successful episodes. The magicwand pipeline can generate the SFT demonstrations from existing chats/files.

When to use RLVR​

How it works​

The rlTask document​

The four checkers​

Creating tasks via the API​

The rl configuration​

Extra packages (pipPackages)​

Model eligibility​

Generation engines​

Monitoring an RL run​

Pros, cons and caveats​