checkLink | Doc Detective

The checkLink action checks if a URL returns an acceptable status code. It primarily issues a browser-mimicking GET request. If the response is still 429 or 403 after retries, it retries the URL once with a HEAD request as a fallback. This action is useful for verifying that a hyperlink or image URL is valid.

You can specify the target URL directly as a string, or use an object for more options:

url: (Required in object format) The URL to check. Can be a full URL or a path. If a path is provided, an origin must be specified either in the step or in the configuration file.
origin: (Optional) Protocol and domain prepended to url when url is a path. If omitted and url is a path, the global origin from the configuration file is used.
statusCodes: (Optional) A single integer or an array of integers representing acceptable HTTP status codes, including non-2xx codes like 429 (rate limited), 403 (forbidden), or 500 (server error). If omitted, defaults to [200, 301, 302, 307, 308].
headers: (Optional) Additional HTTP headers to include in the request. Merged on top of Doc Detective’s default browser-mimicking headers. Accepts either a key/value object or a newline-separated string. Useful for sites that allowlist specific headers or bypass tokens. Defaults to {}.
params: (Optional) Query parameters to append to the resolved URL. Merged with config-level originParams. Values support environment variable substitution with $VAR syntax. Defaults to {}.

Outputs: the action exposes outputs.statusCode, the HTTP status code returned by the request. To capture it for later steps, use the step-level variables object with the expression $$statusCode.

For comprehensive options, see the checkLink reference.

Behavior

Each checkLink step:

Issues a GET request with browser-mimicking default headers (User-Agent, Accept, Accept-Language, Accept-Encoding, Upgrade-Insecure-Requests, and the Sec-Fetch-* and Sec-Ch-Ua* Client Hints set) to reduce false 429 or 403 responses from bot-protection layers.
Retries up to three times (four attempts total) with exponential backoff (1 second, then 2 seconds, then 4 seconds) on 429 or 5xx responses. If the server sends a Retry-After header, Doc Detective waits that duration (capped at 10 seconds) instead.
If the final response is still 429 or 403, retries the URL once with a HEAD request. Some web application firewalls (WAFs) rate-limit HTML GET requests but allow HEAD requests. If the HEAD returns an accepted status code, the step passes.
Compares the final status code against statusCodes and reports PASS or FAIL.

Examples

Here are a few ways you might use the checkLink action:

Check if a link is valid (string shorthand)

{
  "tests": [
    {
      "steps": [
        {
          "description": "Check if Google is up.",
          "checkLink": "https://www.google.com"
        }
      ]
    }
  ]
}

Check if a link is valid (object format)

{
  "tests": [
    {
      "steps": [
        {
          "description": "Check if Google is up.",
          "checkLink": {
            "url": "https://www.google.com"
          }
        }
      ]
    }
  ]
}

Check for a specific status code response

{
  "tests": [
    {
      "steps": [
        {
          "description": "Check if Google is up, expecting only 200.",
          "checkLink": {
            "url": "https://www.google.com",
            "statusCodes": 200
          }
        }
      ]
    }
  ]
}

Check for one of a set of status code responses

{
  "tests": [
    {
      "steps": [
        {
          "description": "Check if Google is up with extra status codes.",
          "checkLink": {
            "url": "https://www.google.com",
            "statusCodes": [200, 201, 202]
          }
        }
      ]
    }
  ]
}

Accept non-2xx status codes

Include non-2xx status codes in statusCodes to accept responses that would otherwise fail. This is useful for testing endpoints that return rate limiting (429), authentication errors (403), or other expected non-success responses.

Doc Detective’s retry-with-backoff and HEAD-fallback logic, described above in Behavior, only runs while the response code isn’t yet accepted. As soon as a response matches statusCodes, Doc Detective stops immediately and passes. That includes a match on the very first GET, as in the example below. It doesn’t retry or fall back to HEAD merely because the code happens to be 429 or 403.

{
  "tests": [
    {
      "steps": [
        {
          "description": "Check API endpoint, accepting rate limit responses.",
          "checkLink": {
            "url": "https://api.example.com/endpoint",
            "statusCodes": [200, 429]
          }
        }
      ]
    }
  ]
}

Check a link with a separate origin and URL path

{
  "tests": [
    {
      "steps": [
        {
          "description": "Check if Google search path is valid using a specific origin.",
          "checkLink": {
            "url": "/search",
            "origin": "https://www.google.com"
          }
        }
      ]
    }
  ]
}

Pass custom headers (allowlist or bypass)

Use the headers field to send additional HTTP headers on top of Doc Detective’s browser-mimicking defaults. This is useful for sites that allowlist a shared secret or require a bypass token.

{
  "tests": [
    {
      "steps": [
        {
          "description": "Check a protected portal with a shared-secret header.",
          "checkLink": {
            "url": "https://portal.example.com/docs",
            "headers": {
              "X-Doc-Detective-Check": "shared-secret"
            }
          }
        }
      ]
    }
  ]
}

headers also accepts a newline-separated string, which is convenient for multi-header strings loaded from environment variables:

{
  "checkLink": {
    "url": "https://api.example.com/endpoint",
    "headers": "X-Api-Key: abc123\nAuthorization: Bearer token"
  }
}

Check a relative link using global origin

Assuming a global origin of https://www.google.com is set in the configuration:

{
  "tests": [
    {
      "steps": [
        {
          "description": "Check if Google search path is valid using global origin.",
          "checkLink": {
            "url": "/search"
          }
        }
      ]
    }
  ]
}

Query parameters

You can append query parameters to a URL using the params option. This is useful for authentication tokens or tracking parameters.

Step-level params: An object specifying query parameters to append to the URL. Applies to all URLs, both relative and absolute. Values support environment variable substitution with $VAR syntax.
Config-level originParams: Default query parameters in your configuration file that apply to relative URLs resolved against origin. Step-level params merge on top, with step keys taking precedence on collision.

When your URL contains a fragment such as /page#section, query parameters are inserted before the fragment, giving /page?token=value#section.

Values in params and originParams are embedded directly in request URLs and appear in test results, logs, and reports. Avoid storing long-lived secrets in these fields.

Check a link with query parameters

{
  "tests": [
    {
      "steps": [
        {
          "description": "Check API health with testing token.",
          "checkLink": {
            "url": "/api/health",
            "origin": "https://my-app.com",
            "params": {
              "__clerk_testing_token": "$CLERK_TESTING_TOKEN"
            }
          }
        }
      ]
    }
  ]
}

Troubleshooting

`checkLink` fails with 429 or 403 even though the URL works in a browser

Sites behind bot protection or a web application firewall (WAF) can return 403 or 429 to non-browser traffic even when the URL works in your browser. Examples include Cloudflare, Akamai, Amazon Web Services (AWS) WAF, Imperva, PerimeterX, and Vercel deployment protection. Doc Detective already sends a browser-like header set and retries transient failures. Some WAFs still fingerprint deeper signals that header spoofing can’t influence, such as Transport Layer Security (TLS) client fingerprints or source IP reputation.

If you hit this, you have three escape hatches:

Widen statusCodes to accept the WAF response code, for example "statusCodes": [200, 301, 302, 307, 308, 403, 429]. This is pragmatic but loses signal, because the check only confirms that the URL exists.

Pass allowlist or bypass headers through the headers field. Common options when you control the site:

Provider	Headers or cookies
Cloudflare Access	`CF-Access-Client-Id` + `CF-Access-Client-Secret` (service-token pair)
Cloudflare Bot Management	`Cookie: cf_clearance=<value>; __cf_bm=<value>` (obtained from a browser session)
AWS CloudFront or Application Load Balancer	Custom allowlist header, for example `X-Origin-Verify: <shared-secret>`
AWS API Gateway	`x-api-key: <key>`
Vercel deployment protection	`x-vercel-protection-bypass: <token>` (optionally `x-vercel-set-bypass-cookie: true`)
Vercel password protection	`Cookie: _vercel_jwt=<value>`
Generic WAF rule	Any custom header and shared secret you allowlist on the WAF

Use a browser-driven step (goTo) instead of checkLink when the URL truly requires JavaScript or a full browser fingerprint.

If you own the site, add a WAF rule that allowlists a custom header and shared secret, such as X-Doc-Detective-Check. Prefer that over trying to mimic a browser.

`checkLink` fails due to unrecognized certificates

The checkLink action might fail for a valid URL that loads without redirects. An internal or custom certificate the testing machine doesn’t recognize is a common cause.

Example

Consider the following test configuration. It checks the validity of https://self-signed.badssl.com/, a website using a self-signed certificate:

bad-certificate.json

{
  "tests": [
    {
      "steps": [
        {
          "description": "Check site with a self-signed certificate",
          "checkLink": {
            "url": "https://self-signed.badssl.com/",
            "statusCodes": [200, 201, 202, 301]
          }
        }
      ]
    }
  ]
}

To run the test, use the following command:

NPX

Docker

npx doc-detective -i bad-certificate.json

This command executes the test, but it fails, returning the following response:

{
  "result": "FAIL",
  "resultDescription": "Invalid or unresolvable URL: https://self-signed.badssl.com/"
}

This occurs because the self-signed certificate isn’t recognized by the testing machine. This behavior is expected in axios, but you can bypass it in Doc Detective by setting an environment variable.

Solution

To fix this issue, follow these steps:

Create a .env file with the following content:
ignore-certificate-problems.env
```
NODE_TLS_REJECT_UNAUTHORIZED=0
```

Modify your test configuration to include a loadVariables step (Note: setVariables is deprecated, use loadVariables):