Teaching a Bot to Play My Zombie Game: a Headless Simulation Lab and an Evolution Strategy

How bots play thousands of matches of LOUD, a Godot zombie roguelite, through the game's own input layer, deterministically and about 30 times faster than real time, and how an evolution strategy trained the best bot's settings and was checked on seeds it never saw. Directed by me, implemented with Claude Code.

20 min read · godot · simulation · evolution-strategies · statistics · claude-code

In short: bots play LOUD, my Godot zombie roguelite, through the same input layer as a human, in the real arena, about 30 times faster than real time and reproducible down to a state hash every second. A batch runner plays about 99 matches a minute and writes statistical reports. An evolution strategy then trained the strongest bot's 23 settings over 11,640 matches. On 900 held-out seed pairs it banked 554 salvage per match against the hand-tuned bot's 120. Claude Code wrote the code; I scoped the work and decided what the results were allowed to mean. For the overview and the key decisions, see the case study. This article is the engineering.

The game, and the question

LOUD is a top-down zombie horde roguelite. Weapons fire on their own, so the player's job is movement, upgrade picks, an aimed BoomBox that lures zombies and then explodes, and one big decision. Loud weapons raise a Heat meter from 0 to 100, Heat raises spawn pressure, and more zombies means more XP and salvage. From 06:00 an extraction beacon opens: hold interact on it for two seconds and you bank the salvage you carry. Stay longer and score multipliers kick in at 08:00 and 10:00, until the run is lost at 12:00. Die and the salvage is gone.

That's a lot of interacting numbers, and nobody had played the latest builds yet. The project has one firm rule: the game is tuned only from playtest evidence. But some questions need more matches than any playtest gives. Does the shotgun cost more health than its damage is worth? Is there a cliff when Brutes arrive at 2:00? What does staying past 06:00 actually earn? Answering those takes hundreds of matches per question, all played the same way, by a player that doesn't get tired.

So the plan was bots. The catch with bots is that it's easy to build a simulation that answers questions about itself. Four rules kept this one honest: bots play the real game, see only what a player sees, produce exactly reproducible matches, and every comparison is paired.

Bots that play the real game

There is no second, simplified simulator. A match is the real arena.tscn with the real spawn director, Heat system, weapons, upgrade drafts and extraction. The only thing a bot replaces is the player's hands. The player reads input through a small object:

class_name PlayerInput
extends RefCounted
## Where the player's hands come from. The default reads the `Input` singleton (keyboard, mouse,
## gamepad). Automated players (`scripts/sim/`) swap in a subclass on `Player.input`, so a bot
## goes through exactly the same gameplay code as a human: movement, the BoomBox throw, the
## beacon hold and the alarm-cache press. Read once per physics tick by each consumer.


## Movement as a stick vector (length 0..1).
func move_vector() -> Vector2:
	return Input.get_vector("move_left", "move_right", "move_up", "move_down")

A bot swaps in a subclass that holds the four things a gamepad can do:

class_name BotInput
extends PlayerInput
## Input written by a bot each physics tick instead of read from devices. Only what a pad can do:
## a stick direction (length ≤ 1), a throw press with a target, an interact press or hold.
## Pulses (`throw`, `interact_press`) last one tick; the bot sets them before consumers read.

var move := Vector2.ZERO
var throw := false
var throw_at := Vector2.ZERO
var interact_press := false
var interact_hold := false

Level-up cards are chosen by pressing the real UpgradeMenu button, so a bot goes through the same menu code a player does.

What a bot may see. A bot gets a SimObservation: the zombies inside the camera view (1280 × 720 plus a 30 px margin), with velocities estimated from two looks; Charger lanes, because the game draws them; pickups and crates in view; and the objectives the HUD arrows point at. It never reads off-screen enemies, spawn schedules or random-number state. A bot that did would rate the game easier than any person could find it. A profile that ever needs more must set "fair": false, and that flag is stored with every match.

How a bot decides. Every three physics ticks (20 times a second) the utility bot scores 17 candidate moves, the 16 compass directions or standing still, and takes the best. This is the core of the scoring loop:

for index in range(-1, DIRECTIONS):
	var d := SimBot.direction(index)
	var p := (obs.me + d * step).clamp(obs.bounds.position + Vector2(obs.radius, obs.radius), obs.bounds.end - Vector2(obs.radius, obs.radius))
	var score := 0.0
	var threat := 0.0
	var nearest := INF
	for k in near.size():
		var i := near[k]
		var gap := p.distance_to(predicted[k]) - obs.enemy_radius[i] - obs.radius
		nearest = minf(nearest, gap)
		var danger := obs.enemy_damage[i] / 10.0
		threat += danger * exp(-maxf(gap, 0.0) / falloff)
		if gap < 0.0:
			threat += danger * 2.0
	score -= threat_w * threat
	if crowd_w > 0.0:
		score -= crowd_w * obs.count_within(p, crowd_r) / 10.0

Zombie positions are pushed forward by their observed velocity to a short horizon, and each one adds danger that falls off with distance. After that come terms for Charger lanes, the arena edge, goals (XP gems, salvage, crates, the beacon) discounted by the danger near them, a preferred distance band for the current weapon, an escape lookahead, open space along the direction, and a little momentum. The BoomBox goes at the densest cluster in range, or away from the bot when it's swamped.

Every weight is a knob in a JSON profile. Seventeen profiles cover a random baseline (which finds crashes and gives a floor), cautious and defensive players, an aggressive one that fights at weapon range and stays for the multiplier, an objective hunter, build specialists, and tactical, the most complete policy. A new behaviour is usually a new profile, not new code.

Fast, and provably the same game

Godot's --fixed-fps 60 makes every rendered frame exactly one 60 Hz physics tick and turns off real-time sync. Headless, the engine then runs ticks as fast as one CPU core allows. Delta is always 1/60, as in a real run; nothing is skipped, scaled, or run at a higher physics rate. A single worker runs a match at about 37 times real time.

"It's the same game, just faster" is a claim, so it's checked. loudsim.py speedcheck plays a set of matches accelerated, then again at normal speed in three ways: headless at real time with the bot deciding live, headless at real time replaying the recorded actions, and in a real window. Every run is compared on a state digest taken every second: the tick, the player's position, HP, XP, level and salvage, kills, the zombie count and the sum of every zombie's position, Heat, pickups and the BoomBox cooldown. On the finished code, 52 of 52 runs matched the accelerated reference, every checkpoint and the final digest. A window renders about 11 frames per physics tick and headless real time about 2.4, so this also shows frame rate doesn't leak into gameplay.

Getting there took two fixes in the game itself, both invisible to a player:

  • Spawn positions read the rendered camera, which carries the screen shake, and the shake isn't seeded. Two identical runs drifted apart the first time something exploded. Spawns now use the camera's target clamped to its limits.
  • Enemy separation broke ties by instance id, which depends on allocation order. It now uses a spawn serial.

The rest is discipline. Every gameplay system has its own seeded RNG stream, bots have theirs, move directions are quantized to 16 and throw targets to whole pixels. Same build, seed and bot profile give the same match, in any process, in any batch order, on any worker count. Any match can be replayed from its recorded inputs, and the replay checks every 10-second checkpoint on the way.

Headless workers also skip presentation: no effect nodes, the world and HUD hidden so nothing draws, no screen shake. Effects use no RNG and the HUD never changes combat, so the digests don't move (a test checks it on every commit), and it bought 17% more throughput.

A lab around it

tools/sim/loudsim.py is the orchestrator, written with the Python standard library only. An experiment is a JSON file:

{
  "name": "heat_decay_ab",
  "seeds": {"start": 2001, "count": 40},
  "factors": {"bot": ["tactical", "aggressive"], "weapon": ["pistol", "shotgun"]},
  "bot_config": {"upgrades": {"type": "random"}},
  "treatments": [
    {"name": "baseline", "overrides": {}},
    {"name": "decay_3", "overrides": {"resources": {"res://resources/tuning/heat.tres": {"base_decay": 3.0}}}}
  ],
  "options": {"timeline_interval": 10, "record_actions": true, "max_wall_seconds": 600}
}

Factors are crossed, and every cell and treatment plays the same seeds, so comparisons pair runs on a seed and seed luck cancels. Treatments change game tuning in the worker's memory only; the resource files on disk are never written. Match ids are a hash of experiment, treatment, cell and seed, which is what makes resuming and joining runs work.

Around that sits the plumbing for unattended runs. Workers report a heartbeat and get killed if they go silent. A crashed or hung worker's unfinished matches go back to the queue, and a match that keeps failing lands in failed.jsonl instead of stopping the run. Ctrl-C or SIGTERM stop cleanly and --resume plays only what's missing. The runner logs CPU, memory and the Mac's thermal limit while it works. Each worker redirects the game's save to a scratch file, and the run fingerprints the player's real save before and after a batch.

On the M4 Pro (10 performance and 4 efficiency cores) the lab measured its own best setup: 14 workers, jobs of at most six matches so no core idles at the end, presentation off. That sustains 99 matches a minute, about 341 hours of game time per real hour, with no thermal throttling over a 1,008-match soak. The remaining bottleneck is single-threaded GDScript: game systems take about 64% of a match, the bot 32%, the telemetry 5%.

report.py turns a run's SQLite database into a standalone HTML report with Measured, Statistical, Hypothesis and Next sections. The statistics are the ones that fit paired, skewed match data: Wilson intervals for rates, bootstrap intervals for means, sign-flip permutation tests on seed pairs with Holm correction, Cliff's delta, and Kaplan–Meier survival with extraction treated as censoring.

What the first studies said

The first balance study ran 1,978 matches: six bots times three starting weapons on 56 shared seeds, a study with randomized card picks, a Heat-decay A/B, and four extraction policies. All valid, and the player's save untouched.

BotExtracted (95% CI)Median deathKills/min
aggressive34% (27–41%)186 s408
objective24% (19–31%)218 s338
tactical17% (12–24%)240 s337
survival2% (1–6%)175 s193
defensive1% (0–4%)202 s237
random0% (0–2%)49 s74

For these bots, offence beats defence: the two policies that run away and take defensive cards almost never reach 06:00, and take more damage per minute than the bots that fight. The shotgun costs about 8 health per minute over the pistol; the bat costs kill speed. Damage taken doubles around 2:30, after Brutes unlock. And two thirds of deaths happened within 150 px of a wall.

That last number led to Experiment 1: are the wall deaths the game, or bad bot navigation?

The bug in the bots' eyes

The experiment added a wall-aware navigation layer: a cost for fleeing straight at a wall, a cost for corners, and a committed breakout toward open ground when the bot gets pinned. It was built and chosen on separate development seeds, then run against the original bots on 100 fresh seeds, 1,800 matches.

It worked on its own terms: deaths near a wall fell by 17 points. It didn't make the bots die less. The deaths moved into the open, and runs got slightly shorter. Wall deaths were a symptom: bots end up at walls because they're being overrun.

The bigger finding surfaced mid-study, in the bots' own perception. The spatial grid that every crowd query goes through was this:

func _build_grid() -> void:
	_grid.clear()
	for i in enemy_pos.size():
		var key := Vector2i(floori(enemy_pos[i].x / GRID), floori(enemy_pos[i].y / GRID))
		if _grid.has(key):
			(_grid[key] as PackedInt32Array).append(i)
		else:
			_grid[key] = PackedInt32Array([i])

In GDScript, a PackedInt32Array read out of a Dictionary is a copy. The append landed on the copy and vanished, so every 96 px cell held only its first zombie. Threat, crowding, the escape lookahead, open-space rays and the BoomBox's cluster search all saw dense crowds as a few scattered zombies.

The fix is three lines, but it went in behind a flag, observation.full_grid, off by default, so every earlier dataset still reproduces digest for digest:

		if not _grid.has(key):
			_grid[key] = PackedInt32Array([i])
		elif full_grid:
			var cell: PackedInt32Array = _grid[key]
			cell.append(i)
			_grid[key] = cell

A post-hoc supplement of another 1,800 matches showed the size of it. Fixing perception alone lengthened runs by 27 seconds, more than anything the wall layer did, and with the fix in place the wall layer added another 15. The verdict on the first study: its directions held up in every variant (offence over defence, the shotgun's health cost, the bat's kill speed, a step at 2:00), but its magnitudes don't, and "Brutes are the main killer" didn't hold up. A bot's result describes the bot.

From hand-tuned to trained

All of this rested on bots whose weights a person had tuned. For tactical, an earlier experiment tried five variants on 20 seeds and kept the best. On its tuning seeds that variant extracted in 30% of runs. On the first study's fresh seeds it managed 17%. Choosing the best of several options on one set of seeds flatters the winner, and that 13-point drop is how much.

The training milestone set out to fix both problems at once: replace the person choosing with an optimizer, and prove the result on seeds nothing ever trained or chose on. The goal in the plan was a sentence that would be true afterwards: the bot's settings were learned by evolutionary search and checked on held-out seeds.

Why an evolution strategy

The bot already decides by weighted scoring, so the most direct thing to learn is the weights. Twenty-three of tactical's settings went into a search space, each with a JSON path, bounds and a type (linear, logarithmic or integer): 17 movement weights, the BoomBox's cluster and panic thresholds, and four extraction settings.

Reinforcement learning with a neural network would have meant a new model, an observation encoding, a deep-learning stack inside a stdlib-only lab, and a policy nobody can read. Searching 23 interpretable numbers needs none of that, and the result is a list of which knobs moved and how far. The cost is a hard ceiling: the search can rebalance what the hand-written scoring already does, but it can't invent a behaviour the bot has no term for.

Measuring the noise first

A match result is very noisy. Fitness is mean salvage banked per match, with 0 on death, because that's what the game asks of a player. But only about one match in six extracts, so per-match salvage had a standard deviation of 200 around a mean of 82. Before committing hours of compute, loudsim.py train-noise measured how well a small seed budget ranks candidates against a 30-seed answer. With 4 seeds per candidate, the better side of a mirrored pair matched the 30-seed answer 76% of the time; with 8, about 88%. I chose 8, doubling the run.

The optimizer

The first optimizer was a smoothed cross-entropy method: sample a population, keep the best quarter, refit. On a synthetic test with 23 knobs, 30 generations and noise at a quarter of the starting gap, it shrank its spread to the floor and stalled, closing 34% of the gap to the optimum. It was replaced by an evolution strategy in the form OpenAI described in 2017: mirrored sampling with centred ranks. On the same test it closed 74%.

Each generation evaluates the current settings plus seven mirrored pairs, a random direction added and subtracted:

def ask(self) -> list[list[float]]:
    rng = random.Random(f"{self.seed}:{self.generation}")
    zs = [[rng.gauss(0.0, 1.0) for _ in range(self.dim)] for _ in range(self.half_population)]
    cands = [list(self.mean)]
    for z in zs:
        for sign in (1.0, -1.0):
            cands.append([_unit(m + sign * self.sigma * zi) for m, zi in zip(self.mean, z)])
    self._pending = (cands, zs)
    return [list(c) for c in cands]

Then it ranks the scores and steps along every direction by how much its plus side beat its minus side:

pert = list(scores[1:])
n = len(pert)
order = sorted(range(n), key=lambda i: (pert[i], i))
rank = [0.0] * n
for r, i in enumerate(order):
    rank[i] = r / (n - 1) - 0.5
step = [0.0] * self.dim
for k, z in enumerate(zs):
    w = rank[2 * k] - rank[2 * k + 1]
    for d in range(self.dim):
        step[d] += w * z[d]
self.mean = [_unit(m + self.lr * self.sigma * s / self.half_population) for m, s in zip(self.mean, step)]

Ranks instead of raw scores mean one lucky extraction can't drag the step, and every sample informs it, not just the top quarter. The search runs in a normalized [0, 1] cube, so a log-scaled knob and an integer knob move on the same footing. Generation g draws from Random(f"{seed}:{g}"), so the whole state is a handful of numbers: a training run can be stopped and resumed and comes out byte for byte the same.

One generation is one ordinary run

The training loop reuses the lab instead of growing a second one. Each generation's 15 candidates become bot_variants of an ordinary experiment, and its seeds become an explicit seed list. All candidates play the same seeds, so the differences between them are theirs and not the seeds'. The seeds change every generation, so the search can't memorise them. One generation is 15 candidates × 3 weapons × 8 seeds = 360 matches.

Three disjoint seed pools. Train, seeds 10000 to 10999, every generation: 8 fresh seeds times 3 weapons times 15 candidates, 360 matches, after which each mirrored pair is ranked and the settings step toward the winners. Validation, seeds 20000 to 20039, every 5 generations: the current settings on the same 120 matches, and the best checkpoint ships. Test, seeds 30000 to 30299, once at the end: the trained and hand-tuned bots on 900 pairs, the only number reported as the result.trainseeds 10000–10999every generation8 fresh seeds × 3 weapons × 15 candidates = 360 matches;rank each mirrored pair, step the settings toward the winnersvalidationseeds 20000–20039every 5 generationsthe current settings on the same 120 matches each time;the best checkpoint is the one that shipstestseeds 30000–30299once, at the endtrained vs hand-tuned bot on 900 seed × weapon pairs;the only number reported as the result
The training run's three seed pools never overlap, and none overlaps any earlier experiment. The test pool was played once.

The seed pools are disjoint from each other and from every earlier experiment. Validation plays the current settings on the same 120 matches every five generations, and the best validation checkpoint is the one kept. The test pool exists to be used once.

The run

The full run went overnight, lid-closed on the MacBook at 14 workers: 30 generations plus seven validations, 11,640 matches in 3 hours 34 minutes, none invalid. A curve.html report rewrote itself after every step, so progress could be checked mid-run.

Salvage banked per match on the 120 validation matches, by training generation: 110 at generation 0, 200 at 5, 336 at 10, 349 at 15, 458 at 20, 474 at 25 and 553 at 30, each with a 95% interval of 40 to 124. On the held-out test seeds the trained bot banked 554 per match and the hand-tuned bot 120.0200400600salvage051015202530generation (validation every 5)testGeneration 0: 110 ± 40 salvage per match, extract 22%Generation 5: 200 ± 52 salvage per match, extract 37%Generation 10: 336 ± 87 salvage per match, extract 42%Generation 15: 349 ± 101 salvage per match, extract 35%Generation 20: 458 ± 119 salvage per match, extract 39%Generation 25: 474 ± 117 salvage per match, extract 40%Generation 30: 553 ± 124 salvage per match, extract 45%110validation: 553Held-out test, trained bot: 554 salvage per match (900 matches)trained554Held-out test, hand-tuned bot: 120 salvage per match (900 matches)hand-tuned120
Salvage banked per match while training: the generation's settings on the same 120 validation matches every five generations, with 95% intervals. Right: both bots on 900 test matches each, played once, after training.
Generation051015202530
Salvage banked per match (validation)110 ± 40200 ± 52336 ± 87349 ± 101458 ± 119474 ± 117553 ± 124
Extraction rate22%37%42%35%39%40%45%

The best checkpoint was the last one, and the curve hadn't flattened.

The held-out test

Validation chose the checkpoint, so its 553 is flattered by that choice. The real number comes from the test pool: the trained profile against the hand-tuned one, paired on 300 unseen seeds × 3 weapons, 900 pairs and 1,800 matches.

hand-tunedtraineddifference (95% CI)
Salvage banked per match120554+434 (+394 to +477)
Extraction rate24.8%48.0%+23 pts (+20 to +27)
Survival272 s354 s+82 s (+74 to +90)
Damage taken per match121121−0.6 (−3.7 to +2.4)
Average Heat6573+8.1 (+7.1 to +9.0)

The trained bot did better in 426 of the 900 pairs and worse in 72; the rest tied. Every salvage and extraction test hit the floor of the permutation test; after Holm correction, p = 0.0016. It was better with every weapon: Pistol 171 → 638, Shotgun 118 → 575, Bat 70 → 448. And validation's 553 against the test's 554 says choosing the checkpoint on validation didn't measurably flatter it.

What it learned

Most of the gain is one decision. The bot's risk extraction policy stays past 06:00 while its health is above risk_hp and the next multiplier step is within risk_wait seconds. Extraction opens at 06:00 and the first step comes at 08:00, 120 seconds later. The hand-tuned risk_wait of 80 meant the bot always left at once. Training pushed it to 200, the top of its range, and dropped risk_hp from 0.55 to 0.24: stay for the multiplier as long as you're not about to die. When the trained bot extracts it banks 1,154 salvage on average, against 483, leaving at 458 s instead of 370 s.

The rest is getting to 06:00 alive. The trained bot died before 06:00 in 411 of 900 matches, against 640, with the same damage taken per match spread over much longer lives and 1.7 times the kills. The movement changes say how: a tighter, less committed orbit around the arena (orbit_radius 0.55 → 0.31, orbit_weight 1.0 → 0.35), more weight on keeping weapon range with a closer near edge (range_weight 0.6 → 1.26, range_lo 0.45 → 0.23), threat that only counts up close (threat_falloff 75 → 38), more crates and gems, and BoomBox throws at clusters of 3 instead of 5.

The knobs moved together, so a single knob's move doesn't prove that knob matters. Ablations would.

What it doesn't show (yet)

  • It learned to play this bot, not to play like a person. It optimises salvage, so it plays greedier and louder (Heat +8). No human playtest data exists for these builds yet, so nothing here is calibrated to people. The game already has a playtest recorder for exactly that comparison.
  • It trained on the old perception. The training run used the default observation, the one that undercounts crowds. A retrain with observation.full_grid on is a new dataset.
  • No ceiling. The curve was still rising and risk_wait sits on its bound. The obvious next run widens it and continues from the best checkpoint.
  • A weak start. The hand-tuned bot extracted in a quarter of runs, so "4.6 times better" says as much about the starting point as about the method.
  • One build. It holds for the default survivor, no contract, three starting weapons and one commit. A gameplay change can invalidate it.

The balance studies keep the hand-tuned bots unless I choose otherwise. The trained bot is a second, stronger reference player: useful for hunting exploits, and for asking what a greedy, competent run looks like.

How I worked with Claude Code

Claude Code (Claude Opus 5.5) wrote the lab, the bots, the optimizer and the reports, ran the batches, and wrote each study up. Every commit carries its co-author line. My job was direction and judgement.

Each phase was a scope I set. The lab itself; Experiment 1, on whether wall deaths were the game or the bots; the throughput work on the M4 Pro; and the training milestone, written as a plan with slices and its defaults listed for me to change before the long run.

The calls were mine. A sample:

  • bot results suggest experiments, but game tuning stays with playtests;
  • eight seeds per candidate over four, after the noise measurement;
  • the hand-tuned bots stay the default for balance studies;
  • the long batches run lid-closed on my MacBook, at the lab's measured worker count.

The reports made reviewing possible. I didn't read every line of the bot. I read reports that kept what was measured apart from what was guessed, said which results held after correction, and listed what would test the rest. The wall study's most useful sentence was the one saying its own earlier numbers were too big.