Skip to navigation
Test your docsUI proceduresRecording

Detect visual regressions in recordings

A recording’s steps can keep passing while the content quietly drifts. Enable checkpoints to compare each step’s frame against a committed baseline, and add verify guards to assert structural properties of the produced video file. New to recording? Start with Record video.

Before you begin

Make sure you have the following:

  • Doc Detective installed.
  • A directory where you can create a .spec.json file.
  • A headed, visible context to capture a fresh recording. Checkpoints and verify guards need a real recording from a visible display. Staleness detection against committed baselines still runs headless. See Detect staleness without re-recording.
  • A committed baseline folder alongside the recording (the first run seeds it).

The steps in this guide work the same in every authoring style. Write them in a standalone .spec.json, inline in a doc source file, or let Doc Detective detect them from prose.

Detect visual regressions with checkpoints

A recording’s steps can keep passing while the recorded content quietly drifts. A page gets redesigned, a button moves, copy changes. To catch that, enable checkpoints on a record step. Doc Detective then captures a screenshot after every step during the span. It compares each against a committed baseline stored beside the recording, at signup-flow.mp4.checkpoints/ by default or an object-form directory:

{
  "record": { "path": "signup-flow.mp4", "checkpoints": true }
}

On the first run the baselines don’t exist yet, so stopRecord seeds them from the captures. Commit the baseline folder alongside the recording. On every later run, each checkpoint compares against its baseline:

  • Within maxVariation (default 0.05): the step passes.
  • Beyond maxVariation: stopRecord returns WARNING, since the recorded flow’s content has probably changed. A comparison never modifies the baselines.

Tune or relocate the comparison with the object form. Raise maxVariation or disable checkpoints for spans that show dynamic content such as animations, ads, and clocks. Those otherwise produce WARNING noise:

{
  "record": {
    "path": "signup-flow.mp4",
    "checkpoints": { "maxVariation": 0.1, "directory": "baselines/signup-flow" }
  }
}

To act on drift instead of only reporting it, set overwrite: "aboveVariation". The span always records, but the new video replaces the existing file only when the span meaningfully changed. That means checkpoint drift beyond maxVariation, an added, removed, or edited step, or a missing recording file. Otherwise Doc Detective discards the fresh capture and leaves the file and baselines untouched, so unchanged recordings stay stable across runs. This mode implies checkpoints. When it refreshes, it updates the baselines with the video in the same step:

{
  "record": { "path": "signup-flow.mp4", "overwrite": "aboveVariation" }
}

Verify the video file itself

Checkpoints validate the scene a recording shows, while verify validates the produced file. Use it to assert structural properties of the video. The classic capture failure modes are a wrong-sized crop, a truncated clip, and an all-black video:

{
  "record": {
    "path": "signup-flow.mp4",
    "verify": { "minDuration": 2, "resolution": true, "notBlack": true }
  }
}

Unlike checkpoint drift, which returns WARNING, a violated verify guard fails the stopRecord step. These are properties you explicitly demand of the artifact:

  • minDuration / maxDuration: the video’s length in seconds.
  • resolution: the video’s dimensions, with a ±2 pixel tolerance because encoders round to even dimensions. true compares against the capture plan’s expectation, or pass { "width": …, "height": … } for literal dimensions.
  • notBlack: fails when black frames cover 95% or more of the video, the symptom of a broken screen capture.

Detect staleness without re-recording

Doc Detective skips the capture in two situations. The first is a headless run, which has no visible display. The second is a target file that already exists under the default overwrite: "false". In both, a span with checkpoints (or aboveVariation, which implies them) still compares its checkpoints against the committed baselines and reports whether the recording has gone stale. stopRecord returns WARNING with stale: true, and a post-run hint points at the fix, which depends on the skip:

  • Headless run: re-run the test on a headed context to refresh the recording.
  • Target already exists: set overwrite to "true" or "aboveVariation", or remove the file.

Detection is cheap and continuous, so headless CI watches recording freshness on every run and you only re-record when something actually changed. Skipped spans never write: no video, no baseline seeding, no baseline updates.

stopRecord reports the outcome of all of this in its outputs: $$checkpoints and $$maxCheckpointVariation, $$seededBaselines, $$changed and $$changeReasons, $$stale, $$duration / $$width / $$height, and $$resolutionMatch / $$allBlack. For the full list, see Recording checkpoints, the stopRecord action reference, and the stopRecord schema.

Verify it works

  • Checkpoint drift beyond maxVariation reports WARNING, so the run continues while flagging that the recorded content probably changed.
  • A violated verify guard reports FAIL, because you demanded that structural property of the file.
  • A skipped capture whose baselines diverged reports WARNING with stale: true.

Troubleshooting

stopRecord returns WARNING with stale: true

Cause: the capture was skipped, from a headless run or an existing file, while the committed baselines diverged from the current content.

Solution: re-run on a headed context to refresh the recording, or set overwrite to "true" or "aboveVariation".

A record step is skipped and the file isn’t refreshed

Cause: the default overwrite: "false" skips a record step whose file already exists.

Solution: set overwrite to "true" to always re-record, or "aboveVariation" to re-record only when the span meaningfully changed.

What’s next