Testing the tests of a basketball simulation: dead mutants, sleeping tests and three answers

Every rule in the simulation has a test, and every test has a planted bug it must catch: a small, deliberate change to the code, such as a flipped comparison or a deleted line, that the test should fail on. If a test still passes with its bug planted, it was not really checking anything. This is called mutation testing. This week the tools that do that checking were themselves checked, and four of them turned out to be quieter than they looked.

Two planted bugs that could never be caught

A planted bug is only useful if the code still runs with it in place. If the change breaks the file's syntax, every test fails at once, for a reason that has nothing to do with the rule under test, and the result says nothing. So the tool now plants each bug in memory and parses the result before running anything, refusing a list that holds one that does not parse. On its first run it found two that had never parsed: one in the rule that runs the game clock while a shot is in the air, one in the camera that follows a player. Both are repaired, and both now fail their tests the way they should.

The browser side had a trap of its own. A JavaScript file can be checked for syntax from the command line, but for a module in a folder that does not declare itself one, that check passed a file with a plain syntax error. The same text, named as a module, failed. The tool now parses each planted script the way the browser will read it.

Two tests that slept for almost four minutes

A full test run sometimes seemed to hang near its end, every processor idle. It was not a hang. Two tests of the step that checks a new version is live waited through that step's real retry schedule, twelve tries ten seconds apart, against a page built to fail. That was 110 seconds each, and at the end of a run they were the only tests left. With the waiting set to zero inside those tests, the file went from 220 seconds to under one. Every test now also has an upper bound of twenty minutes: one that runs past it fails with its own name, instead of looking like a stuck machine.

A source check that could never pass

Every number in the simulation that comes from research names its source, and every source is fetched and pinned by a fingerprint of its bytes, 674 of them. On a fresh copy that check ended at 656 of 674 and reported failure. 17 of the 18 misses were live web pages, such as documentation, that had changed since they were pinned, and one was a server asking for fewer requests. A check that can never pass hides the day something really breaks. It now gives three answers: a real problem fails; a known live page that changed is named but passes, its original copy kept; and a server that refused to answer is reported as not measured, never as fine.

Rules checked across many games

Some rules hold for every moment of every game, such as no player ever standing inside another. They were checked over six fixed games, and one fault that happens about once in 2,200 drives was found only when a new version happened to meet it. One command now checks them over as many games as asked, 24 in under two minutes, before each new version of the engine. Its first finding was not a fault in the players but a test that compared two computed times for exact equality and differed by 0.000000000000002 seconds, a rounding order. It now allows a billionth of a second, like the checks beside it.