# selfplay-chess: a chess ladder that teaches itself This is a small chess engine in pure Python. It learns its evaluation function only by playing itself, and you can upgrade it one knob at a time to get bots of increasing strength. All five piece values start equal (pawn = knight = bishop = rook = queen = 1.0). After about 1000 self-play games of TD-Leaf(λ) on a laptop CPU, the engine has found the usual ordering pawn < knight ≤ bishop < rook < queen by itself. A second stage learns 12 positional weights. A round-robin then turns three knobs into an Elo ladder: - **data**: how many self-play games the weights were trained on - **representation**: material only, or material plus positional features - **test-time compute**: search depth Everything runs on CPU with `numpy`, `matplotlib` and `python-chess`. Pretrained weights are in `weights/`, so `python play.py` works right away. --- ## The levels | Level | What it is | What changed | |---|---|---| | **L0** | random legal move | nothing learned, no search | | **L1** | learned material values, 1-ply greedy (no quiescence) | + **learned evaluation** (stage 1, 1000 games) | | **L2** | same values, depth-3 alpha-beta + quiescence + transposition table | + **search** (test-time compute) | | **L3** | L2 + 12 learned positional weights | + **representation** (stage 2, 300 more games) | | **L4** *(described below, not implemented)* | AlphaZero: deep network + MCTS, trained by self-play | + learned features *and* learned search policy | **L4, AlphaZero (Silver et al., 2018).** AlphaZero replaces our 17 hand-chosen features with a deep residual network. The network takes the raw board and outputs a value (expected result) and a policy (a prior over moves). It also replaces alpha-beta with Monte-Carlo Tree Search (MCTS), which the policy guides. Training is still pure self-play. MCTS visit counts become the policy target, and game outcomes become the value target. So the search improves the policy, and the policy improves the search. Our L1 to L3 are the 1990s version of the same three ingredients (self-play data, an evaluation function, search), with a 17-parameter linear model instead of a network with millions of parameters. That is why it trains in minutes on a CPU, and also why it stays weak: a human club player would beat L3. --- ## The theory, briefly **Evaluation.** `eval(s) = w · f(s)`. Each feature is computed as White's count minus Black's, so the score is from White's point of view and flips sign when the colours are swapped. There are 5 material features (#P, #N, #B, #R, #Q) and 12 positional ones: pawn advancement, passed, doubled and isolated pawns, knight/bishop/queen/king centralisation, mobility, king pawn-shield, bishop pair, and rook on an open file. They are all scaled to be O(1). **Search.** The search is negamax alpha-beta. At the horizon, a quiescence search keeps going through captures and promotions until the position is quiet. If the side to move is in check, it searches every evasion, so mates delivered at the horizon are seen. Moves are ordered as transposition-table move first, then captures by MVV-LVA (most valuable victim, least valuable attacker). Iterative deepening runs under a node or time limit. Repetitions and the 50-move rule count as draws inside the tree. **TD(λ)** (Sutton, 1988) learns to *predict* the game outcome by making each prediction agree with the next one: `d_t = J_{t+1} − J_t`. The last "prediction" is the real result: 1, ½ or 0. Its best-known success is TD-Gammon (Tesauro, 1995). **TD-Leaf(λ)** (Baxter, Tridgell & Weaver, KnightCap, 1998/2000) applies the TD update to the evaluation of the **leaf of the principal variation**, not to the root position. The leaf is the quiet position whose static evaluation the search actually returned. KnightCap learned mainly by playing humans and other programs on an internet chess server. Its authors reported that pure self-play worked less well *for them*, so this project's "self-play only" setup is closer in spirit to **Beal & Smith (1997)**. They used TD learning in self-play to learn chess piece values from an uninformed (equal) start and reported values close to the conventional ones. I have not checked their exact protocol (depth, exploration, number of games), so treat this code as "in the spirit of", not a replication. What `learn.py` does for each game: 1. At every ply, score every legal move with a 1-ply search plus quiescence. Play the best one (ties broken at random). With probability ε = 0.1, play a random move instead. The first 0 to 8 plies of each game are random. 2. For the move actually played, keep the leaf `l_t` of its PV. Set `J_t = σ(β · w·f(l_t))` with β = 0.5. A mate gives J = 0 or 1. This is an on-policy variant: the exploratory moves are part of the trajectory being predicted. 3. After the game, `w ← w + α Σ_t ∇J_t Σ_{j≥t} λ^{j−t} d_j`, with `∇J_t = β J_t(1−J_t) f(l_t)` and λ = 0.7. The learning rate decays as `α/√(1+g/300)`. 4. A game reaching 300 plies is scored as a **draw**. Training never adjudicates by material, because that would leak the textbook values into the learner. Of the 1000 training games, 794 ended in checkmate, which gives the TD signal plenty to work with. Each game updates the weights from both sides at once, because features are always White minus Black. Raw TD weights keep jittering around their fixed point, so the shipped weights are the **average of the last 25 % of checkpoints** (Polyak-style averaging). **Elo.** The Elo fit is a Bradley–Terry / logistic model, fitted by the minorise–maximise algorithm (Hunter, 2004). A draw counts as half a win. Each pair that played gets one virtual drawn game, so a 100 % score still gives a finite rating; this shrinks the gaps a little. L0 is anchored at 0. **Train compute vs test-time compute.** Andy L. Jones, *Scaling Scaling Laws with Board Games* (2021, arXiv:2104.03113), trained AlphaZero agents on Hex at many sizes. Among other results, he found a smooth trade-off: extra train-time compute can replace test-time search, and vice versa. Roughly, each 10× of training compute could replace about 15× of test-time compute in his setting. Our `knob_grid` is a toy, qualitative version of the same question on a much weaker system. Don't read scaling laws into a 7×3 grid. --- ## Install and run ```bash pip install -r requirements.txt # On some Debian/Ubuntu Pythons, building python-chess fails with # "AttributeError: install_layout". This fixed it here: pip install --use-pep517 chess ``` All commands run from this directory. | Command | What it does | Runtime here | |---|---|---| | `python -m pytest` | 29 tests: mate-in-1 and mate-in-2, legality, learning direction, Elo, play loop | **~1.5 s** | | `python play.py` | play White against L2 in the terminal | interactive | | `python play.py --level L3 --color black --export-pgn games.pgn` | play Black against L3, then append the game to a PGN file | interactive | | `python learn.py material --quick` | 120-game stage-1 smoke test (writes to `runs/quick/`) | **~55 s** | | `python learn.py material` | full stage 1: 1000 games, writes `weights/material*.json/png` | **~14 min** | | `python learn.py positional --quick` | 30-game stage-2 smoke test | **~30 s** | | `python learn.py positional` | full stage 2: 300 games with material frozen (`--joint` to also train material) | **~8 min** | | `python ladder.py ladder --quick` | round robin, 2 games per pair | **~25 s** | | `python ladder.py ladder` | round robin, 8 games per pair (`--pgn file` saves all games) | **~3.5 min** | | `python ladder.py grid --quick` | 3 checkpoints × depths {1,3}, 2 games per cell | **~5 s** | | `python ladder.py grid` | 7 checkpoints × depths {1,2,3}, 8 games per cell | **~3.3 min** | | `python ladder.py grid --games-per-cell 24` | the version whose results are in `results/` | **~10 min** | "Here" is a 2-CPU Linux box shared with other jobs, using `--workers 2`. Running the full stage 1 overwrites the shipped `weights/`; pass `--out somewhere/` to keep them. **In `play.py`**, type moves as SAN (`e4`, `Nf3`, `O-O`, `e8=Q`) or UCI (`e2e4`). The commands are `undo`, `moves`, `hint`, `flip`, `help` and `quit`. After each move the bot prints its evaluation in pawns (White's view, or "White mates in N"), the search depth, the node count and its principal variation, for example: ``` L2-material-d3 plays Rg8 eval +0.00 (White's view) depth 3, 1166 nodes, 0.01s PV: 2... Rg8 3. Ng5 Rh8 ``` --- ## Results from my own runs (seed 0; shipped in `weights/` and `results/`) ### Stage 1: piece values from equal starting values (1000 games, 859 s) Raw shipped weights (tail-averaged): P 0.238, N 0.662, B 0.794, R 1.001, Q 1.751. Normalised so that pawn = 1: | games | pawn | knight | bishop | rook | queen | |---|---|---|---|---|---| | 0 | 1 | 1.00 | 1.00 | 1.00 | 1.00 | | 20 | 1 | 1.19 | 1.23 | 1.23 | 1.27 | | 100 | 1 | 1.81 | 2.08 | 2.00 | 2.32 | | 200 | 1 | 2.89 | 2.94 | 3.43 | 4.30 | | 500 | 1 | 2.29 | 2.45 | 3.30 | 5.10 | | 1000 (last iterate) | 1 | 2.72 | 3.49 | 4.22 | 7.65 | | **shipped (avg of last 25 %)** | **1** | **2.79** | **3.34** | **4.21** | **7.37** | | textbook | 1 | 3 | 3 | 5 | 9 | Self-play results: 375 White wins, 419 Black wins, 206 draws. Mean game length was 128 plies. See `weights/material_piece_values.png`. What to notice: - **The ordering emerges fast** (after about 20 games). The *ratios* take hundreds of games and stay noisy. - **The pawn weight is what moves the most.** It drops from 1.0 to about 0.24, while the rook stays near 1. Because everything is divided by the pawn weight, the normalised curves wobble with it. - **Rook and queen come out lower than the textbook values, and the bishop is clearly ahead of the knight.** That is a reasonable answer for *this* player population: shallow, partly random players. - **A second data point:** an earlier 600-game run with a constant learning rate and no averaging ended at 2.27 / 3.24 / 4.16 / 6.20 (last iterate). Same ordering, different numbers. - **A 120-game `--quick` run is too short.** It ended at P 1, N 2.59, B 3.09, R 2.85, Q 3.37, with rook below bishop. ### Stage 2: positional weights (300 games, material frozen, 487 s) These are in pawns per unit of each feature (`weights/positional.json`, `weights/positional_weights.png`): ``` mobility (per 10 squares) +0.83 pawn_advance (per 5 ranks) +0.35 rook_open_file +0.31 passed_pawn +0.30 queen_center +0.10 knight_center +0.05 bishop_pair +0.05 isolated_pawn +0.04 king_center -0.01 doubled_pawn -0.06 bishop_center -0.07 king_shield -0.11 ``` - **Sensible signs:** mobility, advanced and passed pawns, rooks on open files, and doubled pawns. - **Wrong signs, but small:** isolated pawns (+0.04) and king shield (−0.11). These are what the weak self-play population taught, and I am reporting them as learned. - **Centralisation weights are tiny.** Mobility already captures most of what "centralised" means. `weights/positional_heatmaps.png` shows the combined effect as a piece-square table. ### Ladder: round robin, 8 games per pair, random 4-ply openings with colours swapped (208 s) | row vs column | L0 | L1 | L2 | L3 | |---|---|---|---|---| | L0-random | – | 1.5/8 | 0/8 | 0/8 | | L1-material-1ply | 6.5/8 | – | 1.5/8 | 0/8 | | L2-material-d3 | 8/8 | 6.5/8 | – | 1/8 | | L3-positional-d3 | 8/8 | 8/8 | 7/8 | – | **Estimated Elo:** L0 0, L1 +207, L2 +424, L3 +672. How the 48 games ended: 38 checkmates, 4 adjudicated, 3 threefold repetitions, 3 reached the move cap. A game that reaches 200 plies is adjudicated as a win if one side leads by at least 3 pawns in textbook material, and otherwise as a draw. This referee rule is used in tournaments only. The `--quick` ladder (2 games per pair) gave the same ordering: 0 / 159 / 303 / 462. ### Knob grid: training × search depth → score vs a fixed reference (24 games per cell, 606 s) The reference bot is `ref-textbook-d1`: textbook 1/3/3/5/9 values, 1 ply + quiescence. Rows are checkpoints along the whole pipeline: material-only after g games, then positional after 1000 + g games. ``` self-play games d1 d2 d3 0 (material) 0.27 0.29 0.42 20 (material) 0.52 0.54 0.60 100 (material) 0.52 0.50 0.58 1000 (material) 0.52 0.56 0.58 1000+20 (positional) 0.81 0.96 0.94 1000+100 (positional) 0.83 1.00 1.00 1000+300 (positional) 0.85 0.98 1.00 ``` The default run, with 8 games per cell, gave the same picture (`results/` holds the 24-game version). How to read it: - **Search partly substitutes for training.** The untrained engine with all values equal gains 0.27 → 0.42 by searching to depth 3, which closes about 60 % of its gap to the trained engine at depth 1 (0.52). - **The material-only rows go flat after about 20 games.** A deterministic material-only searcher's choices depend mostly on the *ordering* of piece values, and that ordering is learned almost immediately. Many cells share identical games for this reason. - **Representation wins by far here.** Adding positional features, even after only 20 stage-2 games, beats any amount of material training or depth. Depth then turns that into near-100 % scores. - **Noise:** with 24 games per cell, one cell's standard error is up to about ±0.10. --- ## Honest caveats - **Only one training seed is shipped**, plus one earlier run with different settings. Piece-value ratios vary from run to run, and the ordering has been the stable part. - **The pawn weight shrinks a lot during training.** Every number normalised by pawn value is therefore noisy. Compare the raw weights or the tail average, not single checkpoints. - **Self-play learns values for the population it plays in:** 1-ply, ε-random players. The values (bishop > knight, rook ≈ 4.2) and some positional signs reflect that population, not master chess. - **Elo numbers come from 8 games per pair.** They are shrunk by the virtual-draw prior, rely on material adjudication at the move cap, and only mean anything *within this ladder*. They are not FIDE or online ratings. - **The search is plain Python, at roughly 20–60k nodes per second.** L2 and L3 use a 20k-node-per-move cap. L3 sometimes makes horrible moves that any club player would punish. - **What the engine leaves out:** the quiescence search's stand-pat ignores stalemate; there are no null-move, killer or history heuristics; and no opening book. --- ## Ideas for turning this into a tool or demo 1. **Live "watch it learn" demo.** Run `learn.py material` on stage with the piece-value plot redrawing every 20 games, and let the audience guess where the queen will end up. Equal values turning into an ordering within a minute is the hook. 2. **Ladder kiosk.** People play `play.py` against L1, then L2, then L3, and record which level they beat. That gives you a human Elo anchor for the ladder (use `estimate_elo` with humans as extra players). 3. **Knob-twiddling web app.** Wrap `EnginePlayer` in a small Flask/FastAPI server with sliders for checkpoint × depth × feature set, so people can feel the train-vs-test-compute trade-off. `results/knob_grid.png` is the summary slide. 4. **Feature-ablation bandit.** Treat "which positional features to include" or "which depth/time budget" as arms, and use a bandit (UCB or Thompson sampling over match outcomes) to find the best configuration with as few games as possible. This is the sequential-testing problem engine developers solve with SPRT. 5. **Causal question.** The positional weights are a regression on self-generated data. Changing the policy (depth or ε) changes the data distribution and therefore the "value" of a feature. Run stage 2 at depth 1 vs depth 2 and compare: a nice illustration of policy-dependent, off-policy targets. 6. **Swap in a model.** Replace `Evaluator` with a tiny MLP in numpy or PyTorch over the same features, or over a 768-bit board encoding. This is the first step from L3 toward L4. Keep `Searcher` unchanged. 7. **Export to lichess.** The PGNs from `play.py --export-pgn` or `ladder.py ladder --pgn` load directly into lichess's analysis board, so you can show the bots' blunders next to a real engine's evaluation. --- ## Layout ``` selfplay-chess/ ├── engine.py features, linear Evaluator, Searcher (alpha-beta + quiescence + TT), heatmaps ├── learn.py TD-Leaf(λ) self-play training (stage 1 material, stage 2 positional) + plots ├── ladder.py levels L0–L3, game runner, Elo (Bradley–Terry MM), ladder + knob-grid experiments ├── play.py terminal play vs any level, SAN/UCI input, eval + PV display, undo, PGN export ├── requirements.txt ├── pytest.ini ├── tests/ │ ├── test_engine.py mates in 1 and 2, legality, node limit, feature symmetry │ ├── test_learn.py TD update direction/hand-check, masking, tiny self-play run │ └── test_ladder_play.py Elo fit, engine beats random, shipped weights load, scripted play ├── weights/ shipped weights from my runs │ ├── material.json, material_checkpoints.json, material_piece_values.png, material_heatmaps.png │ ├── positional.json, positional_checkpoints.json, positional_weights.png, positional_heatmaps.png │ └── train_material.log, train_positional.log └── results/ ladder_results.json, ladder_elo.png, ladder_games.pgn, grid_results.json, knob_grid.png, grid_run.log ``` `--quick` runs (training and ladder) write to `runs/quick/`, so they never overwrite the shipped `weights/` or `results/`. The full `ladder.py` runs write to `results/`. ## References - R. S. Sutton (1988). Learning to predict by the methods of temporal differences. *Machine Learning* 3, 9–44. - G. Tesauro (1995). Temporal difference learning and TD-Gammon. *Communications of the ACM* 38(3). - D. F. Beal & M. C. Smith (1997). Learning piece values using temporal differences. *ICCA Journal* 20(3). - J. Baxter, A. Tridgell & L. Weaver (2000). Learning to play chess using temporal differences. *Machine Learning* 40(3), 243–263. (KnightCap; an earlier version appeared at ICML 1998.) - D. Silver et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. *Science* 362(6419), 1140–1144. - A. L. Jones (2021). Scaling Scaling Laws with Board Games. arXiv:2104.03113. - D. R. Hunter (2004). MM algorithms for generalized Bradley–Terry models. *Annals of Statistics* 32(1), 384–406.