All files / benches regression-gate.js

100% Statements 82/82
100% Branches 1/1
100% Functions 0/0
100% Lines 82/82

Press n or j to go to the next uncovered block, b, p or k for the previous block.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 831x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x 1x  
/**
 * @file regression-gate.js
 * @description Which benchmarks fail a pull request when they regress, and by
 * how much they may regress first.
 *
 * Read by `scripts/bench-compare.js` in the `benchmark-compare.yml` workflow.
 * Every benchmark is measured and reported on every run; only the ones listed in
 * `gating` can fail the job. The rest are in the comment for a reviewer to read,
 * and a benchmark moves into `gating` once it has shown it can hold still.
 *
 * ## How the numbers are produced
 *
 * Both branches run on the same runner, in the same job, interleaved
 * (base, PR, PR, base, base, PR, ...) so a slow stretch on a shared machine
 * lands on both sides rather than on whichever branch happened to run during
 * it. Each benchmark's figure is the median of its samples across rounds. The
 * comparison is the PR median against the base median; neither absolute number
 * means much on a shared runner, but their ratio does -- the same reasoning
 * `size-check.yml` relies on when it builds both branches side by side.
 *
 * ## Why these thresholds
 *
 * Calibrated by an A/A run: the same commit measured against itself with the
 * workflow's round count, where every difference is noise by construction. A
 * threshold has to sit well clear of the largest A/A difference, or the gate
 * fires on pull requests that changed nothing it measures -- and a gate that
 * fires on everything gets ignored, which is what happened to the original
 * 50 KB bundle ceiling.
 *
 * Calibration run (5 rounds per side, interleaved, one machine; 2026-11-10,
 * i7-1360P, Node 24). A/A median difference, and half the sample range:
 *
 *     dom-patching       -1.8%   ±6%
 *     list-rendering     +5.9%   ±8-17%
 *     reactivity-lookup  +10.2%  ±12%   (samples sit at two levels, ~11 and ~13-14 ms)
 *
 * - **25%** for `dom-patching` and `list-rendering`: four times the largest
 *   A/A difference, which leaves room for a noisier CI runner while still
 *   catching the regressions that matter here. The fallback that re-renders a
 *   whole template instead of patching one binding costs multiples, not
 *   percentages.
 * - **35%** for `reactivity-lookup`. It drifted 10% against itself, so 25%
 *   would leave a margin a shared runner could eat; 35% keeps the same
 *   roughly 3x headroom.
 * - In the same A/A run, `render-paths`, `render-phases`, `render-scenarios`,
 *   `rewind-overhead` and `lru_cache_memory` moved 14-35% against themselves,
 *   and `reactivity`, `router`, `security`, `virtual-list` and
 *   `trace-overhead` showed ±10-40% spread. They report only. Their summary
 *   figure is a microsecond-scale operation that one GC pause moves by a
 *   quarter, and gating on them would need a threshold so wide it catches
 *   nothing. `update-scaling` (-2.1%, ±9%) is the first candidate to add once
 *   CI data confirms it holds still there too.
 * - Build-time benchmarks (`atlas-build`, `incremental-rebuild`,
 *   `component-parser`) are stable but measure the compiler, not an update.
 *   They report only, so the gate stays about what the issue that introduced
 *   it was about: what an update costs.
 *
 * Revisit after a few weeks of real pull-request data: the comment shows each
 * side's spread, and a benchmark whose spread stays small can join `gating`.
 *
 * ## Overriding a failure
 *
 * A maintainer adds the `benchmark-regression-accepted` label to the pull
 * request. The job re-runs, still reports the regression, and passes. Use it
 * for a deliberate trade (a correctness fix that costs time) and say why in
 * the pull request; do not raise a threshold to get one pull request through.
 * @module benches/regression-gate
 */
 
/**
 * Benchmark file name to the largest relative slowdown, in percent, it may show
 * before the job fails.
 * @type {Record<string, number>}
 */
export const gating = {
  'dom-patching.bench.js': 25,
  'list-rendering.bench.js': 25,
  'reactivity-lookup.bench.js': 35,
};
 
/** The label that lets a maintainer accept a reported regression. */
export const overrideLabel = 'benchmark-regression-accepted';