In short: bots play LOUD, my Godot zombie roguelite, through the same input layer as a human, in the real arena, about 30 times faster than real time and reproducible down to a state hash every second. A batch runner plays about 99 matches a minute and writes statistical reports. An evolution strategy then trained the strongest bot's 23 settings over 11,640 matches. On 900 held-out seed pairs it banked 554 salvage per match against the hand-tuned bot's 120. Claude Code wrote the code; I scoped the work and decided what the results were allowed to mean. For the overview and the key decisions, see the case study. This article is the engineering.
The game, and the question
LOUD is a top-down zombie horde roguelite. Weapons fire on their own, so the player's job is movement, upgrade picks, an aimed BoomBox that lures zombies and then explodes, and one big decision. Loud weapons raise a Heat meter from 0 to 100, Heat raises spawn pressure, and more zombies means more XP and salvage. From 06:00 an extraction beacon opens: hold interact on it for two seconds and you bank the salvage you carry. Stay longer and score multipliers kick in at 08:00 and 10:00, until the run is lost at 12:00. Die and the salvage is gone.
That's a lot of interacting numbers, and nobody had played the latest builds yet. The project has one firm rule: the game is tuned only from playtest evidence. But some questions need more matches than any playtest gives. Does the shotgun cost more health than its damage is worth? Is there a cliff when Brutes arrive at 2:00? What does staying past 06:00 actually earn? Answering those takes hundreds of matches per question, all played the same way, by a player that doesn't get tired.
So the plan was bots. The catch with bots is that it's easy to build a simulation that answers questions about itself. Four rules kept this one honest: bots play the real game, see only what a player sees, produce exactly reproducible matches, and every comparison is paired.
Bots that play the real game
There is no second, simplified simulator. A match is the real arena.tscn
with the real spawn director, Heat system, weapons, upgrade drafts and
extraction. The only thing a bot replaces is the player's hands. The player
reads input through a small object:
class_name PlayerInput
extends RefCounted
## Where the player's hands come from. The default reads the `Input` singleton (keyboard, mouse,
## gamepad). Automated players (`scripts/sim/`) swap in a subclass on `Player.input`, so a bot
## goes through exactly the same gameplay code as a human: movement, the BoomBox throw, the
## beacon hold and the alarm-cache press. Read once per physics tick by each consumer.
## Movement as a stick vector (length 0..1).
func move_vector() -> Vector2:
return Input.get_vector("move_left", "move_right", "move_up", "move_down")
A bot swaps in a subclass that holds the four things a gamepad can do:
class_name BotInput
extends PlayerInput
## Input written by a bot each physics tick instead of read from devices. Only what a pad can do:
## a stick direction (length ≤ 1), a throw press with a target, an interact press or hold.
## Pulses (`throw`, `interact_press`) last one tick; the bot sets them before consumers read.
var move := Vector2.ZERO
var throw := false
var throw_at := Vector2.ZERO
var interact_press := false
var interact_hold := false
Level-up cards are chosen by pressing the real UpgradeMenu button, so a bot
goes through the same menu code a player does.
What a bot may see. A bot gets a SimObservation: the zombies inside the
camera view (1280 × 720 plus a 30 px margin), with velocities estimated from
two looks; Charger lanes, because the game draws them; pickups and crates in
view; and the objectives the HUD arrows point at. It never reads off-screen
enemies, spawn schedules or random-number state. A bot that did would rate
the game easier than any person could find it. A profile that ever needs more
must set "fair": false, and that flag is stored with every match.
How a bot decides. Every three physics ticks (20 times a second) the utility bot scores 17 candidate moves, the 16 compass directions or standing still, and takes the best. This is the core of the scoring loop:
for index in range(-1, DIRECTIONS):
var d := SimBot.direction(index)
var p := (obs.me + d * step).clamp(obs.bounds.position + Vector2(obs.radius, obs.radius), obs.bounds.end - Vector2(obs.radius, obs.radius))
var score := 0.0
var threat := 0.0
var nearest := INF
for k in near.size():
var i := near[k]
var gap := p.distance_to(predicted[k]) - obs.enemy_radius[i] - obs.radius
nearest = minf(nearest, gap)
var danger := obs.enemy_damage[i] / 10.0
threat += danger * exp(-maxf(gap, 0.0) / falloff)
if gap < 0.0:
threat += danger * 2.0
score -= threat_w * threat
if crowd_w > 0.0:
score -= crowd_w * obs.count_within(p, crowd_r) / 10.0
Zombie positions are pushed forward by their observed velocity to a short horizon, and each one adds danger that falls off with distance. After that come terms for Charger lanes, the arena edge, goals (XP gems, salvage, crates, the beacon) discounted by the danger near them, a preferred distance band for the current weapon, an escape lookahead, open space along the direction, and a little momentum. The BoomBox goes at the densest cluster in range, or away from the bot when it's swamped.
Every weight is a knob in a JSON profile. Seventeen profiles cover a random
baseline (which finds crashes and gives a floor), cautious and defensive
players, an aggressive one that fights at weapon range and stays for the
multiplier, an objective hunter, build specialists, and tactical, the most
complete policy. A new behaviour is usually a new profile, not new code.
Fast, and provably the same game
Godot's --fixed-fps 60 makes every rendered frame exactly one 60 Hz physics
tick and turns off real-time sync. Headless, the engine then runs ticks as
fast as one CPU core allows. Delta is always 1/60, as in a real run; nothing
is skipped, scaled, or run at a higher physics rate. A single worker runs a
match at about 37 times real time.
"It's the same game, just faster" is a claim, so it's checked.
loudsim.py speedcheck plays a set of matches accelerated, then again at
normal speed in three ways: headless at real time with the bot deciding live,
headless at real time replaying the recorded actions, and in a real window.
Every run is compared on a state digest taken every second: the tick, the
player's position, HP, XP, level and salvage, kills, the zombie count and the
sum of every zombie's position, Heat, pickups and the BoomBox cooldown. On the
finished code, 52 of 52 runs matched the accelerated reference, every
checkpoint and the final digest. A window renders about 11 frames per physics
tick and headless real time about 2.4, so this also shows frame rate doesn't
leak into gameplay.
Getting there took two fixes in the game itself, both invisible to a player:
- Spawn positions read the rendered camera, which carries the screen shake, and the shake isn't seeded. Two identical runs drifted apart the first time something exploded. Spawns now use the camera's target clamped to its limits.
- Enemy separation broke ties by instance id, which depends on allocation order. It now uses a spawn serial.
The rest is discipline. Every gameplay system has its own seeded RNG stream, bots have theirs, move directions are quantized to 16 and throw targets to whole pixels. Same build, seed and bot profile give the same match, in any process, in any batch order, on any worker count. Any match can be replayed from its recorded inputs, and the replay checks every 10-second checkpoint on the way.
Headless workers also skip presentation: no effect nodes, the world and HUD hidden so nothing draws, no screen shake. Effects use no RNG and the HUD never changes combat, so the digests don't move (a test checks it on every commit), and it bought 17% more throughput.
A lab around it
tools/sim/loudsim.py is the orchestrator, written with the Python standard
library only. An experiment is a JSON file:
{
"name": "heat_decay_ab",
"seeds": {"start": 2001, "count": 40},
"factors": {"bot": ["tactical", "aggressive"], "weapon": ["pistol", "shotgun"]},
"bot_config": {"upgrades": {"type": "random"}},
"treatments": [
{"name": "baseline", "overrides": {}},
{"name": "decay_3", "overrides": {"resources": {"res://resources/tuning/heat.tres": {"base_decay": 3.0}}}}
],
"options": {"timeline_interval": 10, "record_actions": true, "max_wall_seconds": 600}
}
Factors are crossed, and every cell and treatment plays the same seeds, so comparisons pair runs on a seed and seed luck cancels. Treatments change game tuning in the worker's memory only; the resource files on disk are never written. Match ids are a hash of experiment, treatment, cell and seed, which is what makes resuming and joining runs work.
Around that sits the plumbing for unattended runs. Workers report a heartbeat
and get killed if they go silent. A crashed or hung worker's unfinished
matches go back to the queue, and a match that keeps failing lands in
failed.jsonl instead of stopping the run. Ctrl-C or SIGTERM stop cleanly and
--resume plays only what's missing. The runner logs CPU, memory and the
Mac's thermal limit while it works. Each worker redirects the game's save to a
scratch file, and the run fingerprints the player's real save before and after
a batch.
On the M4 Pro (10 performance and 4 efficiency cores) the lab measured its own best setup: 14 workers, jobs of at most six matches so no core idles at the end, presentation off. That sustains 99 matches a minute, about 341 hours of game time per real hour, with no thermal throttling over a 1,008-match soak. The remaining bottleneck is single-threaded GDScript: game systems take about 64% of a match, the bot 32%, the telemetry 5%.
report.py turns a run's SQLite database into a standalone HTML report with
Measured, Statistical, Hypothesis and Next sections. The statistics are the
ones that fit paired, skewed match data: Wilson intervals for rates, bootstrap
intervals for means, sign-flip permutation tests on seed pairs with Holm
correction, Cliff's delta, and Kaplan–Meier survival with extraction treated
as censoring.
What the first studies said
The first balance study ran 1,978 matches: six bots times three starting weapons on 56 shared seeds, a study with randomized card picks, a Heat-decay A/B, and four extraction policies. All valid, and the player's save untouched.
| Bot | Extracted (95% CI) | Median death | Kills/min |
|---|---|---|---|
| aggressive | 34% (27–41%) | 186 s | 408 |
| objective | 24% (19–31%) | 218 s | 338 |
| tactical | 17% (12–24%) | 240 s | 337 |
| survival | 2% (1–6%) | 175 s | 193 |
| defensive | 1% (0–4%) | 202 s | 237 |
| random | 0% (0–2%) | 49 s | 74 |
For these bots, offence beats defence: the two policies that run away and take defensive cards almost never reach 06:00, and take more damage per minute than the bots that fight. The shotgun costs about 8 health per minute over the pistol; the bat costs kill speed. Damage taken doubles around 2:30, after Brutes unlock. And two thirds of deaths happened within 150 px of a wall.
That last number led to Experiment 1: are the wall deaths the game, or bad bot navigation?
The bug in the bots' eyes
The experiment added a wall-aware navigation layer: a cost for fleeing straight at a wall, a cost for corners, and a committed breakout toward open ground when the bot gets pinned. It was built and chosen on separate development seeds, then run against the original bots on 100 fresh seeds, 1,800 matches.
It worked on its own terms: deaths near a wall fell by 17 points. It didn't make the bots die less. The deaths moved into the open, and runs got slightly shorter. Wall deaths were a symptom: bots end up at walls because they're being overrun.
The bigger finding surfaced mid-study, in the bots' own perception. The spatial grid that every crowd query goes through was this:
func _build_grid() -> void:
_grid.clear()
for i in enemy_pos.size():
var key := Vector2i(floori(enemy_pos[i].x / GRID), floori(enemy_pos[i].y / GRID))
if _grid.has(key):
(_grid[key] as PackedInt32Array).append(i)
else:
_grid[key] = PackedInt32Array([i])
In GDScript, a PackedInt32Array read out of a Dictionary is a copy. The
append landed on the copy and vanished, so every 96 px cell held only its
first zombie. Threat, crowding, the escape lookahead, open-space rays and the
BoomBox's cluster search all saw dense crowds as a few scattered zombies.
The fix is three lines, but it went in behind a flag, observation.full_grid,
off by default, so every earlier dataset still reproduces digest for digest:
if not _grid.has(key):
_grid[key] = PackedInt32Array([i])
elif full_grid:
var cell: PackedInt32Array = _grid[key]
cell.append(i)
_grid[key] = cell
A post-hoc supplement of another 1,800 matches showed the size of it. Fixing perception alone lengthened runs by 27 seconds, more than anything the wall layer did, and with the fix in place the wall layer added another 15. The verdict on the first study: its directions held up in every variant (offence over defence, the shotgun's health cost, the bat's kill speed, a step at 2:00), but its magnitudes don't, and "Brutes are the main killer" didn't hold up. A bot's result describes the bot.
From hand-tuned to trained
All of this rested on bots whose weights a person had tuned. For tactical,
an earlier experiment tried five variants on 20 seeds and kept the best. On
its tuning seeds that variant extracted in 30% of runs. On the first study's
fresh seeds it managed 17%. Choosing the best of several options on one set of
seeds flatters the winner, and that 13-point drop is how much.
The training milestone set out to fix both problems at once: replace the person choosing with an optimizer, and prove the result on seeds nothing ever trained or chose on. The goal in the plan was a sentence that would be true afterwards: the bot's settings were learned by evolutionary search and checked on held-out seeds.
Why an evolution strategy
The bot already decides by weighted scoring, so the most direct thing to learn
is the weights. Twenty-three of tactical's settings went into a search
space, each with a JSON path, bounds and a type (linear, logarithmic or
integer): 17 movement weights, the BoomBox's cluster and panic thresholds, and
four extraction settings.
Reinforcement learning with a neural network would have meant a new model, an observation encoding, a deep-learning stack inside a stdlib-only lab, and a policy nobody can read. Searching 23 interpretable numbers needs none of that, and the result is a list of which knobs moved and how far. The cost is a hard ceiling: the search can rebalance what the hand-written scoring already does, but it can't invent a behaviour the bot has no term for.
Measuring the noise first
A match result is very noisy. Fitness is mean salvage banked per match, with 0
on death, because that's what the game asks of a player. But only about one
match in six extracts, so per-match salvage had a standard deviation of 200
around a mean of 82. Before committing hours of compute, loudsim.py train-noise measured how well a small seed budget ranks candidates against a
30-seed answer. With 4 seeds per candidate, the better side of a mirrored pair
matched the 30-seed answer 76% of the time; with 8, about 88%. I chose 8,
doubling the run.
The optimizer
The first optimizer was a smoothed cross-entropy method: sample a population, keep the best quarter, refit. On a synthetic test with 23 knobs, 30 generations and noise at a quarter of the starting gap, it shrank its spread to the floor and stalled, closing 34% of the gap to the optimum. It was replaced by an evolution strategy in the form OpenAI described in 2017: mirrored sampling with centred ranks. On the same test it closed 74%.
Each generation evaluates the current settings plus seven mirrored pairs, a random direction added and subtracted:
def ask(self) -> list[list[float]]:
rng = random.Random(f"{self.seed}:{self.generation}")
zs = [[rng.gauss(0.0, 1.0) for _ in range(self.dim)] for _ in range(self.half_population)]
cands = [list(self.mean)]
for z in zs:
for sign in (1.0, -1.0):
cands.append([_unit(m + sign * self.sigma * zi) for m, zi in zip(self.mean, z)])
self._pending = (cands, zs)
return [list(c) for c in cands]
Then it ranks the scores and steps along every direction by how much its plus side beat its minus side:
pert = list(scores[1:])
n = len(pert)
order = sorted(range(n), key=lambda i: (pert[i], i))
rank = [0.0] * n
for r, i in enumerate(order):
rank[i] = r / (n - 1) - 0.5
step = [0.0] * self.dim
for k, z in enumerate(zs):
w = rank[2 * k] - rank[2 * k + 1]
for d in range(self.dim):
step[d] += w * z[d]
self.mean = [_unit(m + self.lr * self.sigma * s / self.half_population) for m, s in zip(self.mean, step)]
Ranks instead of raw scores mean one lucky extraction can't drag the step,
and every sample informs it, not just the top quarter. The search runs in a
normalized [0, 1] cube, so a log-scaled knob and an integer knob move on the
same footing. Generation g draws from Random(f"{seed}:{g}"), so the whole
state is a handful of numbers: a training run can be stopped and resumed and
comes out byte for byte the same.
One generation is one ordinary run
The training loop reuses the lab instead of growing a second one. Each
generation's 15 candidates become bot_variants of an ordinary experiment,
and its seeds become an explicit seed list. All candidates play the same seeds,
so the differences between them are theirs and not the seeds'. The seeds
change every generation, so the search can't memorise them. One generation is
15 candidates × 3 weapons × 8 seeds = 360 matches.
The seed pools are disjoint from each other and from every earlier experiment. Validation plays the current settings on the same 120 matches every five generations, and the best validation checkpoint is the one kept. The test pool exists to be used once.
The run
The full run went overnight, lid-closed on the MacBook at 14 workers: 30
generations plus seven validations, 11,640 matches in 3 hours 34 minutes,
none invalid. A curve.html report rewrote itself after every step, so
progress could be checked mid-run.
| Generation | 0 | 5 | 10 | 15 | 20 | 25 | 30 |
|---|---|---|---|---|---|---|---|
| Salvage banked per match (validation) | 110 ± 40 | 200 ± 52 | 336 ± 87 | 349 ± 101 | 458 ± 119 | 474 ± 117 | 553 ± 124 |
| Extraction rate | 22% | 37% | 42% | 35% | 39% | 40% | 45% |
The best checkpoint was the last one, and the curve hadn't flattened.
The held-out test
Validation chose the checkpoint, so its 553 is flattered by that choice. The real number comes from the test pool: the trained profile against the hand-tuned one, paired on 300 unseen seeds × 3 weapons, 900 pairs and 1,800 matches.
| hand-tuned | trained | difference (95% CI) | |
|---|---|---|---|
| Salvage banked per match | 120 | 554 | +434 (+394 to +477) |
| Extraction rate | 24.8% | 48.0% | +23 pts (+20 to +27) |
| Survival | 272 s | 354 s | +82 s (+74 to +90) |
| Damage taken per match | 121 | 121 | −0.6 (−3.7 to +2.4) |
| Average Heat | 65 | 73 | +8.1 (+7.1 to +9.0) |
The trained bot did better in 426 of the 900 pairs and worse in 72; the rest tied. Every salvage and extraction test hit the floor of the permutation test; after Holm correction, p = 0.0016. It was better with every weapon: Pistol 171 → 638, Shotgun 118 → 575, Bat 70 → 448. And validation's 553 against the test's 554 says choosing the checkpoint on validation didn't measurably flatter it.
What it learned
Most of the gain is one decision. The bot's risk extraction policy stays
past 06:00 while its health is above risk_hp and the next multiplier step is
within risk_wait seconds. Extraction opens at 06:00 and the first step comes
at 08:00, 120 seconds later. The hand-tuned risk_wait of 80 meant the bot
always left at once. Training pushed it to 200, the top of its range, and
dropped risk_hp from 0.55 to 0.24: stay for the multiplier as long as you're
not about to die. When the trained bot extracts it banks 1,154 salvage on
average, against 483, leaving at 458 s instead of 370 s.
The rest is getting to 06:00 alive. The trained bot died before 06:00 in 411
of 900 matches, against 640, with the same damage taken per match spread over
much longer lives and 1.7 times the kills. The movement changes say how: a
tighter, less committed orbit around the arena (orbit_radius 0.55 → 0.31,
orbit_weight 1.0 → 0.35), more weight on keeping weapon range with a closer
near edge (range_weight 0.6 → 1.26, range_lo 0.45 → 0.23), threat that
only counts up close (threat_falloff 75 → 38), more crates and gems, and
BoomBox throws at clusters of 3 instead of 5.
The knobs moved together, so a single knob's move doesn't prove that knob matters. Ablations would.
What it doesn't show (yet)
- It learned to play this bot, not to play like a person. It optimises salvage, so it plays greedier and louder (Heat +8). No human playtest data exists for these builds yet, so nothing here is calibrated to people. The game already has a playtest recorder for exactly that comparison.
- It trained on the old perception. The training run used the default
observation, the one that undercounts crowds. A retrain with
observation.full_gridon is a new dataset. - No ceiling. The curve was still rising and
risk_waitsits on its bound. The obvious next run widens it and continues from the best checkpoint. - A weak start. The hand-tuned bot extracted in a quarter of runs, so "4.6 times better" says as much about the starting point as about the method.
- One build. It holds for the default survivor, no contract, three starting weapons and one commit. A gameplay change can invalidate it.
The balance studies keep the hand-tuned bots unless I choose otherwise. The trained bot is a second, stronger reference player: useful for hunting exploits, and for asking what a greedy, competent run looks like.
How I worked with Claude Code
Claude Code (Claude Opus 5.5) wrote the lab, the bots, the optimizer and the reports, ran the batches, and wrote each study up. Every commit carries its co-author line. My job was direction and judgement.
Each phase was a scope I set. The lab itself; Experiment 1, on whether wall deaths were the game or the bots; the throughput work on the M4 Pro; and the training milestone, written as a plan with slices and its defaults listed for me to change before the long run.
The calls were mine. A sample:
- bot results suggest experiments, but game tuning stays with playtests;
- eight seeds per candidate over four, after the noise measurement;
- the hand-tuned bots stay the default for balance studies;
- the long batches run lid-closed on my MacBook, at the lab's measured worker count.
The reports made reviewing possible. I didn't read every line of the bot. I read reports that kept what was measured apart from what was guessed, said which results held after correction, and listed what would test the rest. The wall study's most useful sentence was the one saying its own earlier numbers were too big.