The Three Tests: Zipf, the Monkey and Predictability

The BPT law

BPT Part 4.10 states four laws used here, each with the book's reading.

• Rank–Frequency Law (G): n_r = n₁ / r^s, "the word ranked r is written 1/r^s as often as the commonest", with s "about 1 in human languages". It is "fitted on the log of count against the log of rank over the words written at least twice".
• Monkey Exponent: s_m = 1 + ln(1 / (1 − p)) / ln 2, "the slope random typing gives, from two letters and the share of silence", with p "the share of the text that is silence". The book derives it: "words of n letters are 2ⁿ in kind and each is written with chance ((1 − p)/2)ⁿ · p, so the slope is ln(2/(1 − p)) / ln 2. With little silence the monkey's slope sits just above 1." So "the slope is read against two controls: the monkey's formula for the universe's own share of silence, and the universe's own text shuffled symbol by symbol."
• Next-Symbol Uncertainty: h_n = ℋ_n − ℋ_k, with k = n − 1: "how uncertain the next symbol is, given the symbols before it", "read on the drive, the written field, row by row, with the silence as a third symbol; n up to 8".
• Structure Over Chance: S = 1 − h_n / h_s, "how much more predictable the text is than itself shuffled": 0 for a text with no order beyond its letter counts, 1 for one that is wholly predictable.

The established physics

Zipf 1949 (Human behavior and the principle of least effort) gives the rank–frequency law. Miller 1957 (Some effects of intermittent silence) shows that random typing gives it too, and Mandelbrot 1953 found its form from the cost of coding. Shannon 1948 (A mathematical theory of communication) gives block entropy, and Shannon 1951 (Prediction and entropy of printed English) gives the next-symbol uncertainty. All are in the BPT sources.

The game's math

Zipf slope (zipfSlope() language.js:72). Take the counts of 2 or more, sorted from most to least, and keep at most the top 500. Fewer than 8 gives no slope. Then, with x = ln(rank) and y = ln(count), fit by least squares:

s  = −Sxy / Sxx
r² = Sxy² / (Sxx · Syy)

The monkey (summary() language.js:203–209). The kept text (more than 64 symbols) is shuffled symbol by symbol, a Fisher–Yates shuffle drawn from mulberry32(seed XOR 0x51ed27). It is split again at the silences, and the new words get their own slope, s_monkey.

Miller's formula (language.js:200–201). p = silences / (letters + silences), and s_Miller = 1 + log₂(1 / (1 − p)), for p < 1. This is BPT's Monkey Exponent, since ln(x)/ln 2 = log₂ x.

Next-symbol uncertainty (entropyRates() language.js:82). For n = 1 to 8, the function slides a window of n symbols, in base 3 since there are three symbols, over the text and takes the block entropy

H(n) = −Σ (c/T) · log₂(c/T)     with T = length − n + 1
h_n  = H(n) − H(n − 1)

It stops early when the text has fewer than 4 · 3ⁿ symbols, too short to estimate blocks of that length. The result is rounded to 4 decimals.

Structure over chance (language.js:211–217). The drive is shuffled with the same generator, continuing its stream, and given its own h. With N the largest n both reached:

S = 1 − h_N(drive) / h_N(shuffled)

Patterned (language.js:241). A text is patterned if S ≥ LISTEN_STRUCTURE = 0.02, or |s − s_monkey| ≥ LISTEN_ZIPF_GAP = 0.1.

Worked case. BPT Part 4.10 reports a level-25 universe in 8.3 that wrote for 636 pulses: 76,396 words, 57,032 distinct, with silence 0.8% of the text. The game's Miller formula gives s = 1 + log₂(1/0.992) = 1 + 0.0116 ≈ 1.01, the value the book prints. The universe's own slope was 1.07 and its shuffled text gave 0.83, a gap of 0.24, which passes 0.1. Its drive was 52% more predictable than its shuffle, and 0.52 passes 0.02. So it was patterned, and the control of Part 7.12 then gave the book's verdict, grammar only.

At level 8 the drive holds 262,144 symbols, and 4 · 3⁸ = 26,244 ≤ 262,144, so all eight block lengths run. Without the stop at 8, n = 9 would also fit (4 · 3⁹ = 78,732). The control's 128 × 128 drive holds 16,384 symbols, and 4 · 3⁸ > 16,384, so the control stops at n = 7. The universe's S and the control's S are therefore read at different n. The verdict compares control with control only (own window against scrambled window, both at n = 7), so it is consistent. The "patterned" test uses the universe's own n.

In the game

The three tests run when the run summary, the Language Block or the record first asks for the language (computeLanguage() app.js:2256). The player sees their results in the block's tiles (app.js:2348–2353):

• "Zipf slope", the universe's s to two decimals, with "monkey x · Miller y" beneath: the shuffled text's slope and the formula's.
• "structure over chance", S as a percentage, with "uncertainty a vs shuffled b" beneath: the last h_n of the drive and of its shuffle.
• "silence", p to three decimals, labeled "share of the field that is null". For a child with a parent in the session, this tile shows the cognate share instead (Part 7.13).

The .txt download prints the same figures in its Tests section (languageBlockText() app.js:2285), and the .json download carries the whole curve of h_n for n = 1 to 8 as predict and predictShuffled.

The scale model

The tests run on the rows read and the drive (Part 7.10), not on the whole field and not at the true level. The slope is fitted over the 500 commonest words, and predictability is read on the 2¹⁸-symbol drive.

Where the game and the theory differ

• BPT fits the rank–frequency law "over the words written at least twice". The game fits it over at most the 500 commonest of those.
• The block's silence tile is labeled "share of the field that is null" (app.js:2352). The number shown is p = silences / (letters + silences) over the written text, one silence per written word, which is BPT's "share of the text that is silence", not a share of the field's cells.

Fixed numbers