Running a training run

Population, generations, checkpoints, and why the fittest genome is not the one you keep

The network is shown eighteen numbers and paid for eating. What is left is running the thing: how many networks, for how long, on what boards, and - the part I got wrong first - which one you keep at the end.

A run

go run ./cmd/snakebatch -train -size 10 -batchSize 100

evolving 150 genomes for 100 generations, seed 7 - ctrl-c stops and keeps the best so far
generation   25 | fitness  13001.2 | best ever  16529.7 | best score  44.42 | best game 75 | nodes  22 | links  58 | species 11
generation   50 | fitness  16746.8 | best ever  19817.1 | best score  61.08 | best game 113 | nodes  22 | links  58 | species  7
generation   75 | fitness  20010.8 | best ever  24896.0 | best score  71.83 | best game 164 | nodes  22 | links  58 | species  9
generation  100 | fitness  26762.1 | best ever  28490.9 | best score  89.83 | best game 232 | nodes  22 | links  58 | species 10

a new network wins: average score 64.88 over 300 games, best 95
saved to demo.json - watch it play with: go run ./cmd/snake -solver neat -neatFile demo.json

A hundred and fifty genomes, a hundred generations, about a minute, and it comes out at 65 of 99. For reference: the safe player - the best thing I wrote by hand - scores 90.8, and it took me a lot longer than a minute.

Fitness is not the number you want

There are two scores on that line and neither of them is the fitness.

Fitness is what evolution sorts on, and it is a made-up number with three terms in it. What anyone actually wants to know is how many fruit are being eaten, so every progress line also carries the best the run has seen in terms of score.

Two figures, because either alone misleads. best game - the highest score of any single evaluation game - is reached early by luck and then never moves. best score - the highest average any genome managed over its twelve games - hides that whole games are being won. Both are kept in the checkpoint, so a resumed run reports the best of the whole run rather than of this sitting.

nodes, links and species are the shape of the population, and they are the early warning system. More on those below.

Every genome gets the same boards

// SharedBoards deals every genome in a generation the same games: the same
// starting positions and the same sequence of fruit, so far as the same
// moves are made. Each generation gets a fresh set.
SharedBoards bool

The idea was to take the luck of the fruit out of comparing two genomes. Measured over sixteen runs each way it made no difference at all to the score - 86.1 against 86.6.

It stays on anyway, for a different reason: it makes a seeded run exactly repeatable, the fruit as well as the breeding. That is what lets a resumed run be checked against an uninterrupted one, and it is what makes every measurement in the last post of this series possible.

Stopping and picking up

A long run has to survive being interrupted, so it saves itself.

// SaveCheckpoint writes the run as it stands. It is written beside the file and
// moved into place afterwards, so a run interrupted mid-write leaves the last
// good checkpoint rather than half of a new one.
func SaveCheckpoint(path string, pop neat.Population, opts Options, record Record) error {

Every twenty-five generations by default, and always at the end. -batchSize is then how many generations this sitting adds, and ctrl-c loses nothing but the generation in progress. The first ctrl-c ends the run cleanly - checkpoint written, best network judged and saved, exactly as if it had finished; a second one is left to kill the process.

There is one setting a resumed run has no choice about. A population grown with connections that loop back will not compile as a feed-forward network, so whether the run is recurrent comes from the checkpoint rather than from whoever picked it up - and the command says so out loud rather than quietly dropping the flag, because a tuning run compared against another one is worth nothing if a flag was silently ignored.

The fittest genome is not the best genome

This is the bit I would not have thought of.

Fitness comes from twelve games. A genome can top the population on the luck of where the fruit landed. So at the end of a run the leading candidates - the best of the last generation, the best of the whole run, and the species elites - are played again over three hundred games each, and the one that holds up wins.

// Pick chooses which network a run keeps.
//
// The fittest genome is not reliably the best one: fitness comes from a handful
// of games, and a genome can top the population on the luck of where the fruit
// landed. So the leading candidates are played again over far more games, which
// is cheap next to training, and the one that holds up wins.

Three things about it that are easy to get wrong and all cost me something:

The candidates are not where you would look for them. By the time a run returns, the generation it scored has already been bred over - pop.Genomes is the next generation and the fitness array has been reset to zeros, so ranking by it sorts on nothing. What is still true is that BestGenome is the best of the last generation actually evaluated, BestEverGenome is the best of the whole run, and the genomes at the front of the new population are the species elites, carried over untouched.

Every candidate gets the same boards, drawn from the run’s seed, so they are compared on the same fruit rather than each on its own luck.

The network already on disk is entered as a candidate too, and so defends its place. Training is noisy enough that a short or unlucky run can come out worse than the last one, and it should not be able to throw away a better network.

the network already saved is better (average score 94.10), so it stays

Watching it

Training in the game window rather than the terminal puts the whole generation on screen at once, every genome playing its own board, and the leader’s network drawn beside it with the readings lighting up as it senses.

It is the clearest picture of what NEAT actually does. It starts as eighteen inputs wired straight to three outputs, and hidden nodes and links appear one at a time as they earn their place.

There is a summary view too, which swaps the wall of boards for a table of numbers - because drawing hundreds of games costs more than running them, and with nothing to draw the run gets the whole machine.

Watching for the population going wrong

Two of the numbers on the progress line are there because I have seen both of them go bad.

Links climbing forever. Mutation adds connections and removes them, and the library’s defaults favour adding four to one. Over six thousand generations that took the best genome from 28 connections to 107 and still climbing, while the score had stopped improving. Bringing the two rates closer together keeps genomes lean enough that a mutation is still a small change.

Species going wrong in both directions. On one run species jumped from 8 to 35 in a few hundred generations and the score fell from the mid thirties to 13 - a population of 150 split 35 ways is four genomes to a species, nearly all of them protected elites, and the search stops searching. Raising the compatibility floor to prevent that overshot completely and pinned the run at a single species, which is no diversity at all.

If the score stalls, look at links and species before you look at anything else.

A note on parallelism

Genomes are scored concurrently, and the obvious thing is to give one run every core. It is not the fastest way to get results.

Parts of a generation are serial, so sixteen workers spend some of their time waiting on each other. Running four seeds at once on four threads each gains about a tenth of the total throughput - and since you need several seeds to conclude anything anyway, you were going to run four of them regardless.

Pros

• A run is one command, saves itself, and resumes exactly.
• Keeping the best-of-300-games rather than the fittest genome is a few dozen lines and it stops a lucky genome walking off with the run.
• The population’s shape is on every progress line, so the two failure modes announce themselves.

Cons

• The knobs interact. Games per genome trades against generations, board sizes trade against wall clock, and none of it is obvious from first principles.
• A hundred generations is a minute and a good network is thousands of them. The feedback loop is slow enough to make bad experimental design expensive.
• Everything above is about running the thing. None of it tells you whether a change you made helped, and that is genuinely harder.

Next: the failure that all of this was hiding.