Speciation

Posted on 16 August 2026

Reading time 7 minutes

Comments

Writing NEAT in GO

golang ai neatgo neat

Measuring how different two genomes are, and why the threshold cannot be a constant

A genome that has just grown a new node is worse than it was. It has an untuned bias, two untuned weights and no idea what the node is for, and in a straight fight with the rest of the population it loses and is gone.

Speciation is the answer: genomes compete mainly against genomes like themselves, so new structure gets a few generations to prove itself.

How different are two genomes?

// CompatibilityDistance measures how different two genomes are.
//
// Genes are matched by their historical marking. Genes present in only one
// genome (disjoint and excess) are normalised by the size of the larger genome
// so that a big genome is not automatically far from everything; small genomes
// are not normalised at all, following the paper. The remaining terms are the
// mean parameter difference over the genes both genomes share.
func CompatibilityDistance(cfg Config, a, b Genome) float64

Three terms:

              unmatched genes           mean weight diff        mean bias diff
distance = c1 --------------- + c2 --------------------- + c3 ----------------
              size of larger              (matching)             (matching)

SpeciesCompatExcessCoeff:     1,
SpeciesCompatBiasDiffCoeff:   .5,
SpeciesCompatWeightDiffCoeff: .5,
SpeciesCompatThreshold:       3,

Structure counts double what parameters do. That is the shape you want: two genomes with the same topology and different weights are variations on a theme, and two with different topologies are different themes.

The normaliser has a special case straight from the paper:

normaliser := a.numGenes()
if b.numGenes() > normaliser {
	normaliser = b.numGenes()
}
if normaliser < 20 {
	// The paper leaves small genomes unnormalised.
	normaliser = 1
}

Dividing by the larger genome is what stops a big genome being automatically far from everything - it has more genes, so it has more chances to hold one the other does not. Below twenty genes the division is skipped, because at that size dividing by the count makes every distance tiny and the population never speciates at all.

Walking two lists

The obvious implementation builds a map of one genome’s genes and looks the other’s up. That is two map constructions per comparison, and every genome is compared against every species representative, every generation.

// geneIndex holds a genome's genes sorted by historical marking.
//
// Sorting is the whole point of an innovation number. Once both genomes are in
// marking order, comparing them is a single walk down the two lists in step,
// with no lookup table to build and nothing to allocate. Building a map per
// comparison instead costs two map constructions for every genome-to-species
// test, which at a few hundred genomes and a dozen species is thousands of
// throwaway maps per generation.

func walkGenes(a, b []gene) (diff float64, matched, unmatched int) {
	i, j := 0, 0
	for i < len(a) && j < len(b) {
		switch {
		case a[i].id == b[j].id:
			if !a[i].fixed {
				diff += math.Abs(a[i].value - b[j].value)
				matched++
			}
			i++
			j++
		case a[i].id < b[j].id:
			// Only a holds this marking.
			unmatched++
			i++
		default:
			// Only b holds this marking.
			unmatched++
			j++
		}
	}
	// Whatever is left in either list is excess.
	unmatched += (len(a) - i) + (len(b) - j)
	return diff, matched, unmatched
}

a:  1   2   3       5   6   7   8
b:  1   2   3   4   5
    =   =   =   b   =   a   a   a
                ^       ^^^^^^^^^
             disjoint     excess

matched 4, unmatched 4, and the walk never went backwards

A merge of two sorted lists, allocating nothing. Speciation indexes every genome and every representative once per generation, and then the thousands of pairwise comparisons are pure walking.

Genes with no value

// gene is one historical marking paired with the parameter it carries: a node's
// bias, or a connection's weight.
//
// An input or bias node has no parameter: it is a fixed signal source that
// every genome in the run shares. It still counts as a gene when matching, but
// its value is left out of the mean parameter difference. Averaging a zero
// difference in for every input would dilute the bias term by however many
// inputs the problem has, so that the same coefficient meant something
// different on every task.

This one took a while to spot. Input nodes are identical in every genome, so their bias difference is always exactly zero - and if those zeros go into the mean, then a problem with 2 inputs and one with 200 have wildly different effective bias coefficients from the same config. A threshold that works on XOR then means nothing at all on anything larger.

They still count as genes for the unmatched term, because they are shared genes and it would be strange for two genomes to look more distant for having sensors in common.

Assigning genomes to species

// Speciate assigns every genome in the population to a species.
//
// Each surviving species keeps a representative drawn from its previous
// members; genomes join the first species whose representative they are
// compatible with, and found a new species otherwise.

First, not nearest. That is the paper’s rule and it is cheaper - a genome that matches species three does not need testing against four through twelve. It does mean species membership depends on the order the species happen to be in, which is a real wart, and it is the same wart the paper has.

The representative is redrawn each generation from the species' members as they were last generation:

// Set species representative to a random member of the generation that
// has just been evaluated, then clear the membership list.
existing.Representative = pop.Genomes[util.RandSliceElement(rng, existing.Genomes)]
existing.Genomes = make([]int, 0)

Keeping the founding member forever would anchor a species to a genome its descendants have long since drifted away from.

Staleness

// A species is stale while its best member fails to beat the best it
// has ever produced.
if bestFitness > s.BestFitness {
	s.BestFitness = bestFitness
	s.Staleness = 0
} else {
	s.Staleness++
}

Against the best it has ever produced, not against last generation. A species that peaked forty generations ago and has been drifting sideways since is stale, even if it wobbles up and down on the way. Selection is what does something about it.

The threshold cannot be a constant

This is the part the paper does not tell you and every implementation ends up discovering.

// AdjustCompatThreshold nudges the compatibility threshold towards whatever
// keeps the number of species near cfg.TargetSpecies.
//
// A fixed threshold is fragile: the same value that yields a healthy handful of
// species at the start will shatter the population into dozens once genomes
// have grown, and once species are down to two or three members each, almost
// every slot goes to elites and the search stops making progress. Retargeting
// the threshold each generation keeps speciation doing its actual job of
// protecting innovation rather than fragmenting the population.
func AdjustCompatThreshold(pop Population) Population {
	if pop.Cfg.TargetSpecies <= 0 || pop.Cfg.SpeciesCompatThresholdAdjust <= 0 {
		return pop
	}
	switch {
	case len(pop.Species) > pop.Cfg.TargetSpecies:
		pop.Cfg.SpeciesCompatThreshold += pop.Cfg.SpeciesCompatThresholdAdjust
	case len(pop.Species) < pop.Cfg.TargetSpecies:
		pop.Cfg.SpeciesCompatThreshold -= pop.Cfg.SpeciesCompatThresholdAdjust
	}
	if pop.Cfg.SpeciesCompatThreshold < pop.Cfg.MinSpeciesCompatThreshold {
		pop.Cfg.SpeciesCompatThreshold = pop.Cfg.MinSpeciesCompatThreshold
	}
	return pop
}

TargetSpecies:                10,
SpeciesCompatThresholdAdjust: .2,
MinSpeciesCompatThreshold:    .5,

Genomes grow. A threshold of 3 that gives eight healthy species at generation ten gives thirty-five at generation five hundred, because there are simply more genes for two genomes to differ on. Thirty-five species in a population of 150 is four genomes each, nearly all of them protected elites, and the search has stopped searching.

So the threshold chases a target species count by a fixed step per generation. Two things about that are worth knowing if you tune it:

It moves slowly. A step of 0.2 a generation means a spike in species count is the threshold failing to keep up rather than the settings being wrong. Reach for the target or the step size before anything else.
The floor is not the knob you want. MinSpeciesCompatThreshold stops the threshold collapsing so far that everything is one species. Raising it to head off a spike overshoots badly and pins the run at a single species, which is no diversity at all.

Next: what the species are actually for.

Jim Wright