Mutation testing with Vitest: a Stryker setup that stays useful

In August I ran mutation testing on the code behind this site for the first time. The suite was green and the coverage looked fine. One afternoon later I knew that three keywords in the prompt-injection filter had never blocked anything.

The filter matches keywords in Dutch and English, and three of the Dutch plurals were written the way boxes? would be in English. That pattern means "boxe" with an optional s. So the singular "box" never matches, and a word that does not exist does. No test had noticed, because no test tried the singular.

A security filter that does not know the singular. Green, of course.

I have written before about why coverage cannot see this. This post is about the setup.

Wiring StrykerJS into a Vitest project takes ten minutes, and that is where most guides stop. You now have a number. Nobody has told you what it counts.

Whether the score means anything comes down to three choices: what you mutate, what you do with static mutants, and which score your threshold checks. I had that last one wrong for months.

Setting up Stryker with Vitest ​

Two packages, next to the Vitest you already have:

npm install --save-dev @stryker-mutator/core @stryker-mutator/vitest-runner

On Vitest 5, take runner 10.1.1 or newer. With an older runner, every mutant survives without a single test running.

The config I ended up with, file names replaced:

// stryker.config.mjs
export default {
  testRunner: 'vitest',
  vitest: { configFile: 'vitest.mutation.config.ts', related: false },
  mutate: ['server/services/search.mjs', 'scripts/audit.mjs'],
  reporters: ['clear-text', 'progress', 'json'],
  jsonReporter: { fileName: 'reports/mutation.json' },
  thresholds: { high: 80, low: 65, break: 65 },
  ignorePatterns: ['releases', '.vitepress/cache'],
  tempDirName: 'node_modules/.stryker-tmp',
}

Three of those lines are there because something broke without them.

• A separate Vitest config. The mutated files are pure services, so the jsdom environment and the Vue plugin from the normal config are dead weight you pay for again on every mutant. The separate config runs in node and lists only the test files that belong to the mutated code. One test that mocks fs/promises at module level did not survive the instrumented run, so it is left out.
• vitest.related: false. By default the runner asks Vitest for only the tests related to the mutated file. With my .mjs tests it found none, and without tests there is nothing to measure. The runner documentation recommends turning it off when your tests do not import the source files directly. Per-test coverage does the real selecting after that.
• ignorePatterns. Stryker copies your working tree into a sandbox. A symlink to a directory made that copy blow up, and copying build output only costs time.

What you can leave out ​

My config had coverageAnalysis: 'perTest', under a comment explaining that the run would never finish without it. The documentation sees it differently:

Your coverageAnalysis property is ignored. The vitest runner plugin will always use "perTest" coverage analysis (which yields the best performance anyway).

So the comment was the only thing that line did. For two and a half months it was the best-documented piece of dead config in the repository.

It is gone.

The same goes for a few Vitest options that Stryker pins itself: one worker per runner, bail on the first failing test, coverage reporting off. Stryker handles parallelism with its own workers, so there is nothing to tune on the Vitest side.

Which files should you mutate? ​

Not all of them. Most of this repository is content and presentation. A surviving mutant in a Vue template: what would you do with it? Write a test proving that a div is a div?

mutate lists six files. They are the places where a test that passes without proving anything has real consequences: code that decides whether a release may go ahead, the search the site's assistant answers from, the prompt-injection filter, and the script that holds the repository to its own rules.

The question per file is simple. If a > quietly becomes a >= here, does anyone notice before a user does? When the answer is "the test", the file belongs on the list.

Static mutants: a quarter of the mutants, half of the time ​

Today's run produced 1,374 mutants. Stryker flagged something straight away: 311 of them, 23%, are static and take an estimated 70% of the test time.

A static mutant sits in code that runs once, when the file loads. A constant at the top of a module, a list of patterns. Stryker cannot switch such a mutant on and off while the tests run, and it cannot tell which tests reach it. So the mutant gets a fresh environment and the whole suite.

In my case they are mostly strings and regular expressions: 139 and 104 of the 311. Those are the pattern lists at the top of the injection filter and the search code.

--ignoreStatic skips them. I measured both on the same code:

Run times depend on what else the machine is doing, so I timed the two back to back, twice: 1:58 against 1:00, then 1:41 against 0:45. Stryker's estimate was 70%. It came out at half. Estimates can be measured too.

Half the time gone, and the score barely moves. Tempting.

Except the three broken keywords from the opening lived in exactly that kind of static list. Skipped mutants get the status Ignored and do not count, including when something is wrong with them.

Then there is the noise. Same code, same tests, four full runs today, four different scores: from 67.98 to 69.32 on total. Across three of those reports, 79 mutants changed status from one run to the next, and 65 of them were static.

With ignoreStatic, three runs stayed within 0.2 of each other. Why the static ones wobble, I have not found out. That the instrument has a margin of its own, I now know.

My rule: ignoreStatic for the quick run while working, the full run for the measurement that counts, and a threshold with some room under it.

Two scores, one threshold ​

Stryker calculates two scores and prints them side by side:

• Mutation score: killed (or timed out) divided by all valid mutants. The total column in the report.
• Mutation score based on covered code: killed divided by the mutants at least one test reaches. The covered column.

The difference is the NoCoverage mutants: code no test comes near. Today there were 178 of those, next to 930 killed and 260 survived. That makes 78.15 on covered and 67.98 on total.

thresholds.break checks the mutation score, which is the total one. When it drops below, Stryker exits with code 1.

In August I set break to 75, "from the measurement rather than from a wish", as I proudly noted at the time. The measurement was 80.04. On covered. Total stood at 71.16 that day, so the threshold was red from the moment I set it.

It did come from a measurement, just the wrong one.

It now sits at 65, against 67.98 on today's lowest run. And I steer by total. Covered is the number you want to believe, because it strikes out exactly the code that is worst off. What no test reaches, no test can guard.

Which survivors do you let go? ​

A score under 100 is a to-do list, and not every item is worth doing.

• Regex mutants. 69 of today's 260 survivors. Stryker rewrites the inside of a pattern in ways that still match every realistic input. Chase them all and you end up writing strings that exist only to fail a mutated regex, which is testing the tool. One thing I do keep: one test case per alternative in a pattern, which is how the missing singulars surfaced.
• Equivalent mutants. Two guards in a clean-up function cannot fail, because the step before them already fixes the outcome. They get a note in the test file. Inventing a scenario to kill a mutant that cannot break anything is occupational therapy.
• Code that only starts a process. The function that calls npm run build has nothing but uncovered mutants. Testing it means a real build per mutant, or a fake process that pins down nothing that matters.

A score of 100% on these files would mean the tests were written for the mutator.

Where does it run? ​

In August it was 631 mutants and 40 seconds. Today it is 1,374 mutants and somewhere between 1:41 and 3:20, depending on what else the machine is doing. The count more than doubled, mostly because the search code has grown since. Mutation testing grows with your code, and faster than your suite does.

That is too much per commit, certainly beyond six files. What works is a scheduled run, weekly or nightly, with the JSON report kept so you can hold the previous score next to it. A run like that belongs with the guards that can go red, and the threshold under it should only ever get stricter.

Coverage was happy with everything in August. The mutation run found three words the filter did not know, in an afternoon. The setup is the easy part. The number arrives by itself after that, and what it measures is what you chose.