Reporters and artifacts

Every Doc Detective run produces output. That means a summary in your terminal, machine-readable results, and, when your tests capture them, screenshots and recordings. This guide covers choosing which reporters run, where each one writes its output, and how to fail a CI build.

For the full list of reporter and output options, see the config reference.

Before you begin

• Install Doc Detective and have a spec you can run.
• Know your output directory. Reporters that write files use the output config option (or the --output flag); it defaults to the current working directory.

Step 1: Choose your reporters

Doc Detective ships six reporters: terminal, json, html, runFolder, junit, and markdown. By default, it runs terminal, json, and runFolder. To choose a different combination, set the reporters option.

The junit and markdown reporters write a fixed filename in the output directory and overwrite it on each run, so a CI artifact path stays stable. If output names a file, they write beside that file.

On GitLab, point the test summary at the junit.xml file:

doc-tests:
  script:
    - doc-detective --input ./docs --reporters terminal junit --output ./reports
  artifacts:
    reports:
      junit: reports/junit.xml

On GitHub Actions, append the Markdown summary to the job summary:

- run: doc-detective --input ./docs --reporters terminal markdown --output ./reports
- if: always()
  run: cat reports/doc-detective-summary.md >> "$GITHUB_STEP_SUMMARY"

Select reporters on the command line:

doc-detective --input api.spec.json --reporters terminal json html

Or set them in your config file so every run uses the same combination:

{
  "reporters": ["terminal", "json", "html"]
}

Step 2: Read the terminal summary

The terminal reporter prints a summary grouped by specs, tests, contexts, and steps, with counts for passed, failed, warnings, and skipped. When everything passes and at least one test ran, it prints a success banner. When no tests run, whether from an empty input or an active filter, it prints a warning instead. An empty run therefore never looks like a green run.

Step 3: Find the JSON results

The json reporter writes the full results to a file:

• If output ends in .json, Doc Detective writes to that path, adding a numeric suffix (-0.json, -1.json, …) to avoid overwriting an existing file.
• Otherwise it writes testResults-<timestamp>.json in the output directory.

The results file carries a summary object with the pass, fail, warning, and skipped counts. The per-spec, per-test, per-context, and per-step detail follows it. For an example of the results structure, see Create your first test. To act on results programmatically, parse this file. See Fail CI when tests fail.

Find slow specs with durationMs

Every node in the results carries a durationMs field in whole milliseconds. That covers the run, each spec, each test, each context, and each step. Doc Detective populates it. You never set it yourself.

{
  "durationMs": 61840,
  "specs": [
    {
      "specId": "api.spec.json",
      "durationMs": 48200,
      "tests": [{ "testId": "create-a-widget", "durationMs": 48200 }]
    }
  ]
}

The meaning shifts slightly by level, and the difference matters when you run tests concurrently:

Spec and test durations are sums of work, not elapsed spans. Contexts from every test run through the same set of concurrent runners. An elapsed span for a spec would therefore count time while unrelated specs were running. Summing keeps the number stable no matter what you set concurrentRunners to. That’s what you want when comparing specs to each other.

The consequence: when concurrentRunners is greater than 1, the run’s durationMs is less than the sum of its specs’ durations, because specs overlapped. That’s expected. Read the run value as elapsed wall-clock time, and the spec values as where the work went.

To find the specs slowing your CI down, sort the specs by durationMs:

jq '.specs | sort_by(-.durationMs) | .[:5] | .[] | {specId, durationMs}' testResults-*.json

Anything that runs more than once reports the final attempt in durationMs. That’s the attempt the reported result describes. What that costs you depends on which kind of retry happened:

• A step re-run by onFail retry routing reports only its last attempt. The enclosing context’s duration still covers every attempt and the waits between them, so the time isn’t lost from the totals. It isn’t separable, though. The difference between a context and the sum of its steps also includes ordinary setup and cleanup. Read it as an upper limit on what retrying cost, not a measurement of it. Use attempts to tell whether a step retried at all.
• A context re-run by the retries policy reports only its last attempt. No context, test, or spec duration absorbs the earlier ones. The run-level durationMs is wall-clock, so it does include them. The retries count is your signal that it happened.

A step that never ran reports 0. That covers a step that was skipped, guarded off, or downstream of a failure.

Read the warm-phase timings

The results also carry a warm object that records what the run provisioned before executing tests. At the start of every run, Doc Detective front-loads setup the tests would otherwise pay one at a time. It runs those tasks concurrently. That setup covers browser and driver installs, simulator and emulator boots, and related downloads. Each entry lists the task, its outcome, and how long it took:

"warm": {
  "durationMs": 4211,
  "tasks": [
    { "name": "browser-install:chrome", "kind": "browser-install", "outcome": "skipped", "durationMs": 2, "note": "'chrome' is already available" },
    { "name": "session-probe", "kind": "session-probe", "outcome": "warmed", "durationMs": 4205, "note": "1 combination ok" }
  ]
}

Outcomes are warmed (Doc Detective performed the work), skipped (nothing to do, or the environment handles it later), and failed. The warm phase is best-effort. A failed task logs a warning and never fails the run. The affected test context retries or skips with its normal behavior and reason. Use the timings to spot provisioning costs in slow CI runs. A long warmed task on every run usually means the runtime cache isn’t persisted between runs.

Step 4: Find the HTML report

The html reporter writes a self-contained HTML report you can open in a browser:

• If output ends in .html or .htm, Doc Detective writes to that path, adding a numeric suffix (-0.html, -1.html, …) to avoid overwriting an existing file.
• Otherwise it treats output as a directory and writes testResults-<timestamp>.html there (or coverageResults-<timestamp>.html for a coverage run).

The report embeds the same results the json reporter produces. A single file is therefore enough to share a run with someone who doesn’t have Doc Detective installed.

Step 5: Collect the run folder and artifacts

The runFolder reporter archives each run into its own folder so you can attach it to a CI job or diff one run against another:

<output>/.doc-detective/runs/<runId>/
├── testResults.json
├── testResults.html
└── specs/<specId>/
    └── tests/<testId>/
        └── contexts/<contextId>/
            ├── screenshots/<NN>-<action>-<stepId>.png
            └── recordings/<contextId>.mp4

The file names inside the folder are constant, because the run ID lives in the folder name. Two run folders therefore compare cleanly. Auto-captured screenshots and recordings land beside the results in the same folder. The whole folder becomes a single artifact to upload from CI for reviewers to inspect.

Step 6: Fail CI when tests fail

This is the most common mistake when wiring Doc Detective into a pipeline, so read it carefully.

To gate your build, choose one of these approaches:

• Set --exit-on-fail (least setup, works on any CI). Pass the flag, or set exitOnFail: true in your config. The CLI then exits with code 1 whenever a spec fails, so the job fails like a broken unit test. WARNING results are non-fatal and never trigger a non-zero exit. It works the same everywhere, including GitLab CI, Jenkins, CircleCI, and a bare shell.

doc-detective --input ./docs --exit-on-fail

• Use the GitHub Action. Set its exit_on_fail input to true so the job fails when tests fail. See the GitHub Action guide.
• Parse the JSON results. Read the summary object from the json reporter’s output and fail the step yourself when any fail count is greater than zero. Useful when you want custom logic beyond a pass/fail gate.

doc-detective --input ./docs --reporters json --output results.json && \
  node -e "process.exit(require('./results.json').summary.specs.fail > 0 ? 1 : 0)"

The && runs the check only if Doc Detective itself exited cleanly. A crash or a broken config still fails the step, and the results parse never hides it.

Verify it works

After a run, confirm that:

• The terminal shows the summary you expect.
• The json reporter wrote a results file at the path you configured.
• If you enabled runFolder, a .doc-detective/runs/<runId>/ folder exists with testResults.json and any captured screenshots or recordings.

What’s next

• Run Doc Detective in CI with the GitHub Action.
• Run in Docker and headless for containerized pipelines.
• Troubleshoot a failing or flaky test when the results aren’t green.