# Robot Sumo: self-play on a laptop CPU

Two disc-shaped, differential-drive robots in a circular ring, like the mini
robot-sumo contests at maker fairs. A robot loses the moment its center leaves
the ring. The physics is a small vectorized numpy simulator: traction-limited
wheels, sideways wheel grip, motor lag, soft contact, sensor noise and delay.
Policies are small neural nets trained **without backprop** by evolution
strategies (ES). The levels go from a random bot, to a hand-coded charger, to a
net trained against the charger, to self-play against a league of its own past
versions, to self-play under domain randomization. Everything trains on 2
laptop CPU cores in about half an hour (a 2.5-minute `--quick` smoke run is
also provided), and pretrained weights are included, so `watch.py` and
`evaluate.py` work right away.

![L3 vs the charger](media/L3_vs_L1_normal.gif)

## Quick start

```bash
pip install -r requirements.txt            # numpy, matplotlib (+ pytest for tests)

python watch.py --p0 L4 --p1 L2 --save l4_vs_l2.gif     # render a match to GIF (~3 s)
python watch.py --p0 L4 --p1 L1 --perturb icy --live    # watch in a window
python watch.py --human L1                              # YOU vs the charger (arrows/WASD, R restart, Q quit)
python evaluate.py                                      # tables + plots for the shipped weights (~1.5-2 min)
python evaluate.py --quick                              # same, 100 matches/cell (~25 s)
python -m pytest -q tests                               # 14 tests, ~8-20 s

python train.py --quick       # smoke-test the whole pipeline into weights_quick/ (~2.5 min)
python train.py               # full retrain into weights/ (~29 min here; overwrites shipped weights)
python train.py --stage L4    # just one stage (it uses the previous stage's weights)
```

Use any level name with `--p0/--p1/--human`: `L0`, `L1`, `Dodger`, `L2`, `L3`,
`L4`, or an intermediate self-play checkpoint such as `L3_gen0040`.
`--perturb` takes one of `normal icy heavy_opp laggy noisy`. MP4 output
(`--save x.mp4`) needs ffmpeg; GIF output only needs Pillow, which comes with matplotlib.

## The levels

| Level | What it is | How it's made |
|---|---|---|
| **L0** | random wanderer | holds a random (thrust, turn) for 0.5 s |
| **L1** | "charger" | hand-coded: turn to face opponent, full throttle |
| **Dodger** | "matador" | hand-coded: sidestep when targeted, retreat from the edge, hit from the side |
| **L2** | MLP 16→32→32→2 (tanh), 1,666 weights | ES vs the charger only (120 generations) |
| **L3** | same net | ES self-play vs a growing pool of its own checkpoints + hand-coded bots (200 gens, starting from L2) |
| **L4** | same net | L3 continued with domain randomization of floor friction, masses, motor lag, sensor noise and delay (150 gens) |

## The theory, briefly

**Evolution strategies** (Salimans et al. 2017, *Evolution Strategies as a
Scalable Alternative to Reinforcement Learning*, arXiv:1703.03864). Smooth the
non-differentiable match outcome F(θ) with Gaussian noise. The gradient of
J(θ) = E_ε[F(θ + σε)] is (1/σ)·E[F(θ + σε) ε]. That gradient can be estimated
just by playing matches with perturbed weights, so the simulator never needs
to be differentiable. We use the standard tricks from that paper:
*antithetic pairs* (+ε and −ε, which reduce variance), *centered-rank fitness
shaping* (only the ordering of fitnesses matters, which makes the update robust
to outliers and to the scale of the reward), weight decay, and Adam on the
estimated gradient. All 48 perturbed policies play the *same* 32 start
positions and opponents in each generation (common random numbers). They also
run in one vectorized simulator call of about 1,500 matches.

**Self-play and the league.** Training only against the current version of
yourself tends to cycle (rock-paper-scissors dynamics), and a policy can
"forget" how to beat older strategies. Fictitious play (Brown 1951) and its
learning version, fictitious self-play (Heinrich, Lanctot & Silver, ICML 2015),
play against the *historical mixture* of past strategies instead. L3 does a
light version of this. Every 20 generations a snapshot joins the pool. Opponents
are 30% hand-coded bots and 70% pool members. Of the pool picks, 40% are drawn
from the 3 newest snapshots. The rest use *prioritized fictitious self-play*
(PFSP, from AlphaStar: Vinyals et al., *Nature* 2019): we keep a running
win-rate estimate against each snapshot and sample opponents with weight
∝ (1 − win rate)², which focuses training on the opponents we still lose to.

**Domain randomization** (Tobin et al., IROS 2017, for vision; Peng et al., ICRA
2018, for dynamics). Train on a *distribution* of simulators so the real world
looks like just another sample. In L4, 75% of training matches draw new physics
per match: friction μ ∈ [0.15, 1.0] (log-uniform), each robot's mass ∈ [0.7,
2.2] kg, motor lag τ ∈ [20, 350] ms, sensor noise σ ∈ [0, 0.2] (on 60% of
robots), and observation delay of 0–2 control steps (0–100 ms).

**Reward shaping (disclosed).** Final evaluation is only win / loss / tie.
During training:
* L2: a win counts up to 0.3 more the faster it happens, and an early loss
  costs up to 0.3 more. At the time cap, a tie gives up to ±0.3 for ending
  closer to the ring center than the opponent.
* L3: the same shaping at 0.15, decayed to zero by generation 150 (the last 50
  generations are pure win/loss/tie).
* L4: the same shaping at 0.1, plus a **tie penalty of 0.25**. Without it, the
  first L4 run (pure win/loss/tie, every match randomized, slightly wider
  ranges) learned to *hide and run out the clock*. Against the charger on a normal floor it went from
  99% wins to about 5% wins and 90% ties. Under randomized physics, attacking
  is risky (on ice, a charge can carry you out of the ring), so a 0 for a tie
  looked safe. This is a real effect that's worth showing.

## Physics model (sumo_env.py)

Per robot: position, velocity, heading, turn rate, plus the lagged motor states.
Action (thrust, turn) ∈ [−1, 1]². Each 50 ms control step runs 5 semi-implicit
Euler substeps:

* Motor commands pass through a first-order lag (τ, 50 ms nominal).
* Wheel force = 12 N·thrust minus longitudinal drag along the heading, plus a
  strong lateral force that cancels sideways sliding. Their **vector sum is
  capped by the friction circle μ·m·g**. So on a normal floor, pushing power is
  traction-limited, a heavier robot pushes harder, and on ice robots skid.
* The turn rate follows the lagged turn command (max 6 rad/s). Turning
  authority also shrinks when μ < 0.5, which is a simplification.
* Robot–robot contact is a spring-damper (k = 3000 N/m, c = 15 N·s/m) between
  frictionless discs.
* Ring radius is 1 m, robot radius 0.1 m, and an episode is capped at 12 s
  (then it's a tie). A simultaneous exit is also a tie.

Observations (16 numbers) are **egocentric and role-symmetric**: the opponent's
position, relative velocity and relative heading in my body frame; my velocity,
turn rate and motor states; the vector to the ring center; my distance and the
opponent's distance from the center; and the time left. Because the same
function computes both robots' observations, one policy can play either side.
The tests check swap symmetry and invariance to rotating the arena.

## Results from my run (shipped weights, `python evaluate.py`, 400 matches per cell)

These are real numbers from this machine (2 CPUs, other jobs running at the same
time). Score = win + ½·tie, so 0.5 means even. A different seed will give
different numbers.

**Round robin (row player vs column player):**

| row vs col | L0 | L1 | Dodger | L2 | L3 | L4 |
|---|---|---|---|---|---|---|
| **L0** | 0.54 | 0.04 | 0.03 | 0.10 | 0.01 | 0.00 |
| **L1** | 0.95 | 0.49 | 0.66 | 0.00 | 0.01 | 0.04 |
| **Dodger** | 0.94 | 0.34 | 0.50 | 0.01 | 0.04 | 0.07 |
| **L2** | 0.89 | 1.00 | 0.99 | 0.50 | **0.00** | **0.00** |
| **L3** | 0.98 | 0.99 | 0.95 | 1.00 | 0.50 | 0.50 |
| **L4** | 1.00 | 0.96 | 0.94 | 1.00 | 0.50 | 0.50 |

Mean score against the others: L0 0.04, L1 0.33, Dodger 0.28, L2 0.58, **L3
0.89, L4 0.88**.

**Robustness.** The level plays robot 0 against a fixed, unhandicapped
opponent. `icy` sets μ = 0.2 for both robots. `heavy_opp` gives the opponent
2 kg instead of 1 kg. `laggy` sets *our* τ to 300 ms and adds 100 ms of
observation delay. `noisy` adds σ = 0.15 sensor noise and 50 ms of delay to us.

| vs **L1 charger** | normal | icy | heavy_opp | laggy | noisy | mean |
|---|---|---|---|---|---|---|
| L2 | 1.00 | 0.05 | 0.49 | 0.19 | 0.91 | 0.53 |
| L3 | 0.98 | 0.33 | 0.29 | 0.04 | 0.50 | 0.43 |
| L4 | 0.94 | **0.64** | 0.49 | 0.13 | 0.58 | **0.56** |

| vs **Dodger** | normal | icy | heavy_opp | laggy | noisy | mean |
|---|---|---|---|---|---|---|
| L2 | 0.98 | 0.15 | 0.50 | 0.13 | 0.90 | 0.53 |
| L3 | 0.94 | 0.60 | 0.50 | 0.02 | 0.61 | 0.53 |
| L4 | 0.93 | **0.82** | 0.48 | 0.15 | 0.70 | **0.62** |

| vs **L3** | normal | icy | heavy_opp | laggy | noisy | mean |
|---|---|---|---|---|---|---|
| L2 | 0.00 | 0.01 | 0.00 | 0.00 | 0.00 | 0.00 |
| L3 | 0.50 | 0.51 | 0.50 | **0.00** | 0.47 | 0.40 |
| L4 | 0.50 | 0.53 | 0.57 | **0.19** | 0.50 | **0.46** |

The full tables, including hand-coded rows, are printed by `evaluate.py` and
saved to `results/eval.json`. The plots are in `plots/`:
`winrate_matrix.png`, `robustness.png` and `training_curves.png`.

### What actually happened (vs. the story we hoped for)

* **L2 exploits the charger and is brittle against learners: yes.** It beats
  the charger 100% of the time and, unexpectedly, also beats the Dodger 98%
  of the time. Against both self-play levels it loses **every one of 400
  matches**. It also collapses on ice (0.05 vs the charger).
* **Self-play is broadly stronger: yes.** L3 beats every earlier level at
  0.95 or above. While L3 trained, its score against L2 went from 0.5 (all
  ties) to 0.99 between generations 20 and 40, and it held up against the hand-coded bots
  (`plots/training_curves.png`).
* **Domain randomization helps under perturbations: partly.** L4 has the best
  mean robustness score against all three fixed opponents. The clearest gain
  is on ice: 0.33 → 0.64 vs the charger and 0.60 → 0.82 vs the Dodger. When
  handicapped with laggy motors against L3, L4 still scores 0.19, where L3 ends
  up losing to itself 100% of the time. The cost is a little nominal
  performance (0.98 → 0.94 vs the charger). **Laggy motors remain hard for
  every level**, and against a heavier opponent everything mostly draws,
  because you can't push 2 kg out with 1 kg of traction.
* **The strongest learners draw each other.** L3 vs L3, L3 vs L4 and L4 vs L4
  are 100% ties. They circle each other near the center, and neither one gives
  up its side (see `media/L3_vs_L4_stalemate_2x.gif`). That matches the game
  theory: a symmetric zero-sum game has value 0, and self-play has found
  strategies that guarantee roughly that against each other. Only a
  perturbation breaks the deadlock.
* There is a caveat on intermediate checkpoints: matches between learned bots
  are almost deterministic (the only randomness is the start position), so
  small gaps between nearby checkpoints can look like 0% or 100%.

## Runtime on this machine (2 CPUs, shared with other jobs, Python 3.13, numpy 2.5)

| Command | Time |
|---|---|
| `python train.py` (full: L2 120 gens + L3 200 + L4 150) | L2 533 s + L3 705 s + L4 492 s ≈ **29 min** |
| `python train.py --quick` | 2 min 32 s (smoke test only: quick-mode L3/L4 are weak) |
| `python evaluate.py` / `--quick` | 105 s / 24 s |
| `python watch.py ... --save x.gif` | ~2.5 s for a short match, longer for a full 12 s draw |
| `python -m pytest -q tests` | 8–21 s |

One generation (48 policies × 32 matches, up to 240 control steps × 5
substeps) takes about 2.5–4 s. Training runs in a single process.

## How this connects to real robotics

* **Simulate, then transfer.** Real robot learning increasingly trains
  entirely in a physics simulator and deploys "zero-shot." The two tricks here
  are the main ones used for that: **domain randomization** (so the policy
  doesn't overfit one set of physics) and **modeling actuator lag, sensor noise
  and latency**. These unmodeled effects are a classic cause of sim-to-real
  failure. Our `laggy` result is a small example: 300 ms of motor lag destroys
  policies that never saw it.
* **Self-play for physical competition.** Bansal et al. (ICLR 2018, *Emergent
  Complexity via Multi-Agent Competition*) trained simulated MuJoCo humanoids
  to sumo-wrestle with self-play and an opponent pool. Our project is a
  2-D, CPU-sized cousin of that. Google DeepMind's soccer work (Haarnoja et al.,
  *Learning agile soccer skills for a bipedal robot with deep reinforcement
  learning*, Science Robotics 2024) trained small humanoid robots in
  simulation. As I understand it, the pipeline first trained skills and then
  combined them through self-play against earlier versions of the agent, with
  randomization and perturbations so the policy transferred to the real robots
  without fine-tuning. Check the paper for the exact recipe.
* **Next step with a GPU: MuJoCo Playground** (Google DeepMind, 2025). It is a
  GPU-accelerated collection of MuJoCo/MJX environments for locomotion and
  manipulation, with training recipes that are meant to go sim-to-real in a
  short time. My understanding is that it pairs with JAX-based PPO training.
  The ideas here (vectorized envs, randomization, self-play pools) carry over
  directly. You would swap ES for PPO and the 2-D discs for 3-D robots.
* Why ES and not PPO here: ES needs only forward passes, it parallelizes
  trivially and it ignores sparse or delayed rewards. That makes it perfect for
  a numpy-only, CPU-only demo. It is less sample-efficient than policy-gradient
  methods for big networks.

## Ideas for turning this into a tool or demo

* **Booth demo:** `watch.py --human L4` on a laptop. Visitors try to beat each
  level in turn (an L0→L4 "ladder"), and a leaderboard records who got furthest.
* **Live perturbation slider:** extend `watch.py` with matplotlib sliders for
  μ, mass, lag and noise, and show the learned policy failing or adapting in
  real time. This makes the case for domain randomization in 30 seconds.
* **"Train in front of the audience":** run `train.py --quick` on a projector
  with the training-curve plot updating live.
* **Hardware bridge:** the action space (thrust, turn) maps directly onto a
  two-wheel robot (e.g. a micro:bit or ESP32 car). The 16-number observation
  could come from an overhead camera tracking two colored markers. Export
  `weights/L4.npz` as three matrices; the forward pass is about 1,700
  multiply-adds, which fits a microcontroller.
* **Research hooks:** causal questions such as "which randomized factor buys
  which robustness?" (ablate one DR dimension at a time and fit a factorial
  model). You could also treat league sampling as a bandit problem (PFSP is
  essentially a heuristic opponent-selection bandit), compute Nash mixtures
  over the checkpoint pool from the payoff matrix (a meta-game, à la PSRO), or
  add asymmetric robots so the game is no longer value 0.

## File layout

```
robot-sumo/
├── README.md
├── requirements.txt
├── sumo_env.py        # vectorized simulator, physics params, perturbations, observations
├── policies.py        # L0 random, L1 charger, Dodger, MLP (batched), level registry, match runner
├── train.py           # ES (antithetic, centered ranks, Adam), PFSP league, L2 -> L3 -> L4
├── evaluate.py        # round-robin matrix, robustness tables, plots, results/eval.json
├── watch.py           # GIF/MP4 replay, live window, human-vs-bot keyboard mode
├── tests/test_sumo.py # physics sanity, symmetry, ES improvement, tool smoke tests
├── weights/           # shipped: L2, L3, L4 + L3_gen*/L4_gen* self-play checkpoints, training logs
├── plots/             # winrate_matrix.png, robustness.png, training_curves.png
├── results/eval.json  # raw W/L/T for every cell
└── media/             # example match GIFs
```

## Honest caveats

* The physics is a toy: frictionless discs, no wheel-level model, no ring
  edge lip, and a hand-tuned turning penalty on ice. It is meant to be
  qualitatively right (traction-limited pushing, skidding, lag) and is not
  calibrated to any real robot.
* Results come from **one training seed**. ES and self-play are noisy, so
  expect different numbers (and possibly different stories) with another seed.
* The Dodger heuristic is weak: it loses to the plain charger. It is there
  mostly as a behaviorally different opponent.
* The tie penalty and shaping in L4 were added *after* seeing the first L4 run
  go passive. This is an honest example of reward design, not a pre-registered
  experiment.
* The `--quick` mode only checks that the pipeline runs. Its L3/L4 are much
  weaker than the shipped ones.