Building SimReach: Behavior Cloning, PPO, and VLA on a Planar Arm

I built SimReach as a portfolio simulation demo for comparing robot learning pipelines on a small task. A two-link arm reaches the red or blue marker named by an instruction. Everything runs on CPU, and the scenes are schematic drawings. This is a way to inspect demonstrations, training, and evaluation together, with no claim about physical robot performance.

The useful question here is how much the data and observation format change the result. I generate clean, noisy, and deliberately mislabeled demonstrations, audit them against the simulator, and train behavior cloning and vision-language-action policies. PPO supplies an online learning baseline. The repo includes measured results from one run, so there is something concrete to inspect before rerunning it.

Keep the task small enough to audit

The environment in simreach/env.py uses two 1 m links, a 0.1 second time step, and at most 50 actions per episode. Each action contains two normalized joint velocities in [-1, 1]. The simulator scales those commands to joint velocities and integrates the angles.

The vocabulary has exactly two instructions: reach the red target and reach the blue target. Success means getting within 0.12 m of the instructed marker. This limited vocabulary matters when interpreting the language models below. I am testing target selection in a controlled scene, with no evidence here of understanding arbitrary requests.

Reward comes from simreach/task.py:

    dist = distance_to_goal(q, red, blue, instruction_id)
    success = dist < success_threshold
    # Dense hill plus a terminal bonus. The hill gives PPO a direction; the bonus
    # makes actually reaching the target worth more than hovering nearby.
    reward = float(np.exp(-3.0 * dist))
    if success:
        reward += float(success_bonus)
    return reward, float(dist), bool(success)

The success bonus defaults to 5. A successful action ends the episode, while a timeout truncates it. I keep success and final distance alongside return because accumulating the dense reward for many steps can make a near miss look good.

The pipeline and its observation boundary

The main path is expert rollouts, HDF5 storage, validation, training, then evaluation on shared seeds. simreach/benchmark.py connects those pieces and writes checkpoints, JSON reports, plots, and GIFs.

BC and PPO receive a 12-value privileged state. It contains joint sine and cosine features, joint velocities, end-effector position, the instructed target offset, and an instruction one-hot. The actual feature builder uses the selected goal offset, rather than separate coordinates for both markers.

The vision policies receive an image, six joint features, and the instruction. They do not receive target coordinates. Stored training frames show the arm and both markers without highlighting the instructed one. The white ring and caption in the saved GIFs are added for readers. That boundary makes the image task harder, and it prevents treating the state and vision rows as an equal-information algorithm comparison.

Generate errors with a known meaning

The scripted expert in simreach/expert.py uses damped least-squares inverse kinematics. It converts end-effector error into a bounded Cartesian velocity, solves through the Jacobian, and normalizes the joint command.

Gold demonstrations follow the instructed target. Noisy demonstrations corrupt that controller with random actions or scaled expert actions plus Gaussian noise. Mislabeled demonstrations use the clean controller toward the opposite color. The target selection in simreach/dataset.py is explicit:

    match quality:
        case Quality.GOLD | Quality.NOISY:
            tracked = int(env.instruction_id)
        case Quality.MISLABELED:
            tracked = other_color(int(env.instruction_id))
        case _:
            assert_never(quality)

I store states and images with length T+1, including the terminal state, while actions and rewards have length T. The HDF5 file has a meta/info.json sidecar with LeRobot-style feature names. This is an HDF5 representation, rather than a complete LeRobot export.

Dataset seeds start at 10,000 for the default seed, leaving the evaluation seeds separate. The three tiers use distinct episodes, so the tier comparison also includes differences in sampled scenes and dataset sizes.

Recompute labels before trusting them

simreach/validate.py checks shapes, instruction consistency, action bounds, and image content. It then recomputes every joint update, reward, and success flag from the stored trajectory. Kinematic and reward tolerances are 1e-4.

Gold has additional requirements: it must succeed, track the instructed color, remain sufficiently smooth, and finish closer to that target. Structural failures fail the dataset regardless of tier. Deliberate negative controls can have quality warnings without failing the overall report.

The checked-in results/validation.json records 360 episodes and 10,673 transitions, with zero structural failures and all 200 gold episodes passing. It also flags two mislabeled trajectories that touched the instructed target. I want that distinction visible: a valid file can still contain intentionally bad supervision.

Start imitation with a small state policy

simreach/bc.py standardizes the observations and trains an MLP against expert actions using mean squared error. Its network is straightforward:

        self.net = nn.Sequential(
            nn.Linear(obs_dim, hidden),
            nn.ReLU(),
            nn.Linear(hidden, hidden),
            nn.ReLU(),
            nn.Linear(hidden, act_dim),
            nn.Tanh(),
        )

The final Tanh bounds the commands. Checkpoints include the standardizer, so inference applies the same transformation as training. The benchmark uses 50 epochs, hidden width 64, weight decay 0.001, and observation noise 0.05. Those are run settings, not general recommendations.

Add images without hiding the language shortcut

The tiny policy in simreach/vla.py trains a convolutional image encoder and a two-entry instruction embedding on demonstrations. Its constructor makes the limited text representation clear:

        self.vision = TinyVisionEncoder(out_dim=128)
        self.text = nn.Embedding(2, 64)
        self.head = ActionHead(self.vision.out_dim, 64, hidden=hidden)

The action head concatenates visual features, text features, and encoded proprioception. The optional CLIP path instead computes frozen pretrained image and text embeddings and trains an action head on them. simreach/clip_encoder.py freezes the backbone explicitly:

        self.model.eval()
        for parameter in self.model.parameters():
            parameter.requires_grad_(False)

That path downloads openai/clip-vit-base-patch32 on first use. It is an action head trained on frozen features, with no fine-tuning of the CLIP backbone.

PPO in simreach/ppo_train.py uses Stable Baselines3, a CPU MLP policy, and online environment interaction. The reported budget is 49,152 steps. It learns from the same privileged state as BC, without demonstrations.

Run it and read the artifacts

Use Python 3.10 or newer. These commands are copied from the README; the editable install includes the declared dependencies:

pip install -e ".[dev]"
pytest
python -m simreach benchmark --out results --with-clip

Omit --with-clip to skip the pretrained encoder download and its result row. Running the benchmark into results regenerates artifacts there, so preserve the checked-in files first if you want to compare runs. Separate CLI commands in the README also expose generation, validation, BC, VLA, PPO, and evaluation.

The existing metrics file records seed 0, 64-pixel images, 200 gold demonstrations, 80 noisy, 80 mislabeled, 25 VLA epochs, and 40 evaluation episodes. Every method uses evaluation seeds 0 through 39.

In that run, gold BC succeeds on 70% of episodes, versus 2.5% for noisy BC and 7.5% for mislabeled BC. Gold tiny VLA reaches 17.5%, frozen CLIP reaches 20%, PPO reaches 10%, and the expert reaches 100%. These are single-run measurements, without uncertainty estimates.

Gold BC has mean return 11.97 versus the expert's 8.67, despite lower success. Longer episodes can accumulate more dense reward. I read success and final distance first, rather than calling the higher return a better controller.

Tests, limits, and the next experiment

GitHub Actions installs CPU dependencies on Ubuntu with Python 3.11 and runs pytest. Tests check the Jacobian against finite differences, deterministic resets, integration, expert reaching, HDF5 round-trips, and rejection of broken kinematics. Learning tests check loss changes and action shapes or bounds. CLIP uses a stub encoder in tests, so CI does not download the real checkpoint.

The quick benchmark test uses seven demonstrations and two evaluation episodes. It checks that the pipeline writes its artifacts, not that it reproduces the published scores.

My next experiment would repeat training across seeds and report variability before increasing model size. I would also match tier sizes and scenes to isolate corruption more carefully. Physical dynamics, camera variation, and broader language remain outside this demo's evidence.

Explore the repo

The SimReach repository contains the simulator, dataset specification, tests, and saved results. I would start with the validator and one generated trajectory, then run the full benchmark. That makes it easier to understand what each policy learned and what the score actually measures.