Before any of the algorithm, a handful of helpers for drawing random numbers and poking at slices. They are eight short functions and it would be fair to skip them - except that two of them encode decisions that everything else in the series depends on.
Every one of them takes the random source
func FloatBetween(rng *rand.Rand, min, max float64) float64
func Chance(rng *rand.Rand, p float64) bool
func Gaussian(rng *rand.Rand) float64
func RandSliceElement[T any](rng *rand.Rand, s []T) T
That first parameter is the whole reproducibility story, and it is enforced by the signature rather than by remembering.
The version of these I wrote first did not have it. They called the package-level math/rand functions, which is shorter, reads better, and quietly makes a run impossible to repeat: the sequence depends on process-wide state that anything else in the program can also draw from.
// Rand is the random source driving a run.
//
// Evolution is stochastic, so nothing about a run is repeatable unless every
// random choice comes from a source the caller controls. Reaching for the
// package-level math/rand functions instead leaves the sequence at the mercy of
// process-wide state, which makes a promising result impossible to reproduce
// and a rare crash impossible to bisect.
type Rand = rand.Rand
It only takes one call site forgetting. A single rand.Float64() somewhere in mutation and the whole run is unrepeatable, with nothing to show for it - the same seed produces a different answer and you have no idea which of two hundred lines is responsible. Making the source an argument means there is no shorter way to write it.
Everything downstream leans on this. Seeded runs, snapshot-and-resume, and the test that asserts the same seed gives the same population at 1, 2, 4, 8, 16 and unlimited workers are all impossible without it.
Chance
// Chance reports whether an event with probability p occurs.
func Chance(rng *rand.Rand, p float64) bool {
if p <= 0 {
return false
}
if p >= 1 {
return true
}
return rng.Float64() < p
}
Almost every mutation starts with a call to this. The two guards are worth having: a rate of zero means genuinely never rather than “never, unless Float64 returns exactly zero”, and a rate of one means always. Configuration turns features off by setting a rate to zero, so “off” needs to mean off.
They also skip the draw entirely, which keeps a disabled feature from consuming the random stream. Turning a mutation off then changes only that mutation, rather than shifting every subsequent random number in the run.
Gaussian
// Gaussian returns a sample from the standard normal distribution.
//
// The standard library's ziggurat implementation is several times faster than
// the Marsaglia polar method: no rejection loop, and no log or square root on
// the common path. Weight and bias perturbation calls this for nearly every
// gene of every offspring, so it sits squarely in the hot path of a generation.
func Gaussian(rng *rand.Rand) float64 {
return rng.NormFloat64()
}
A one-line wrapper that exists to be a decision rather than a habit. Writing Box-Muller or Marsaglia polar by hand is the reflex, and NormFloat64 is both faster and one line.
Why a gaussian at all, rather than a uniform draw: perturbation wants most steps to be small and the occasional one to be large. A uniform step is as likely to be a big disruption as a small refinement, which is the wrong shape for tuning something that is nearly right.
Clamp
// Clamp constrains v to the inclusive range [min, max].
func Clamp(v, min, max float64) float64 {
if v < min {
return min
}
if v > max {
return max
}
return v
}
Every weight and bias goes through this on its way out of a mutation, against MinWeight/MaxWeight and MinBias/MaxBias. Without it a run of unlucky perturbations walks a weight out to a few thousand, the activation saturates, and the gene is stuck: every subsequent perturbation is lost in the noise of a number that large.
Two ways to remove from a slice
Here is the pair that matters.
// RemoveSliceIndex removes the element at index i. The order of the remaining
// elements is not preserved.
func RemoveSliceIndex[T any](s []T, i int) []T {
s[i] = s[len(s)-1]
return s[:len(s)-1]
}
// RemoveSliceIndexOrdered removes the element at index i, preserving the order
// of the remaining elements.
func RemoveSliceIndexOrdered[T any](s []T, i int) []T {
return append(s[:i], s[i+1:]...)
}
The first is the one everybody writes. Swap the last element into the hole, shorten by one, constant time, no shuffling. When you do not care about order it is strictly better.
The first version of this library used it to delete a node from a layer. Nothing in it uses it now:
genome.Layers[nodeToDelete.layer] = util.RemoveSliceIndexOrdered(genome.Layers[nodeToDelete.layer], nodeToDelete.nodeIndex)
A layer is not an unordered bag. The order of nodes within it is the evaluation order, and for a recurrent network that decides which connections run forwards this pass and which read the previous activation:
layer: [ h0 h1 h2 h3 ] h1 -> h3 runs forwards
h3 -> h1 reads the last step
delete h2, unordered:
layer: [ h0 h1 h3 ] unchanged, by luck
delete h0, unordered:
layer: [ h3 h1 h2 ] h1 -> h3 now reads the last step
h3 -> h1 now runs forwards
both loops reversed, by deleting
a node that is in neither of them
Deleting one node silently rewires the memory of connections at the other end of the layer. Nothing errors, the genome is still valid, and the network it compiles to is a different network than it was a moment ago for reasons that have nothing to do with the mutation.
The ordered version is O(n) in a slice of a handful of nodes, a few times in a thousand offspring. That is nothing, and it buys a guarantee that a genome’s shape is a function of its history rather than of which gene happened to be deleted last.
The fast one is still in the package, unused, for the next time something genuinely does not care.
Taking a random element
// RandSliceElement returns a random element of s. It panics if s is empty.
func RandSliceElement[T any](rng *rand.Rand, s []T) T {
if len(s) == 0 {
panic("cannot take a random element of an empty slice")
}
if len(s) == 1 {
return s[0]
}
return s[rng.IntN(len(s))]
}
The panic is deliberate. The alternative - returning the zero value - hands back a Genome{} with no layers and no connections, which is a perfectly valid-looking struct that fails much later and somewhere else entirely. Every caller here has already established the slice is non-empty; if one has not, it is a bug and I would rather know at the line that caused it.
The single-element case skips the draw for the same reason Chance does: a species of one should not consume a random number that a species of two would.
They are internal now
internal/util/
In the first version these were an exported util package. They are implementation details, and exporting them promises they will not change - which for a generic slice helper is a promise worth nothing to anybody and a nuisance to me.
If you want the random helpers, math/rand/v2 has them. What is worth taking from this package is not the code, it is the argument for that first parameter.
Next: the part that actually runs a network.