runShell

The runShell action runs a shell command or script on the local machine and evaluates the results. That extends Doc Detective’s testing to anything you can script.

runShell runs the command in bash by default on every platform, Windows included, so the same command behaves the same everywhere. To use a Windows shell instead, set the shell property. See Shell selection.

Even with the consistent bash default, embedding a code snippet in a runShell command still sends it through the shell’s quoting and escaping rules. The nested quotes and parentheses in a one-liner like node -e "..." are a common source of errors. For a JavaScript, Python, or Bash snippet with nested quotes, parentheses, or multiple lines, use runCode instead. It writes the code to a temporary file and runs it directly, so there’s no shell-quoting step to trip over.

You can specify the command directly as a string or use an object for more options:

• String Shorthand: Provide the command string directly as the value for the runShell key.
• Object Format: Use an object with the following properties:
  • command: (Required) The command or script to execute.
  • shell: (Optional) The shell to run the command in: bash (default), cmd, or powershell. cmd and powershell only work on Windows. If unset, uses the config-level shell setting, which defaults to bash.
  • args: (Optional) An array of arguments to pass to the command.
  • workingDirectory: (Optional) The directory in which to run the command.
  • timeout: (Optional) Maximum duration in milliseconds to wait for the command to complete.
  • exitCodes: (Optional) An array of acceptable exit codes. If the command’s exit code is not in this list, the step fails (default: [0]).
  • stdio: (Optional) A string or regular expression to validate against the command’s combined stdout and stderr. If the output doesn’t match, the step fails. Regex must start and end with / (for example, /^hello world.*/).
  • background: (Optional) An object that starts the command as a long-running background process. The step returns as soon as the process is ready, instead of waiting for it to exit. Including this object enables background mode. Omit it for a normal blocking run. See Background processes.
    • name: (Required) A unique identifier for the background process within the run. Reference it from a closeSurface step to stop the process, or from a type step to send it keystrokes.
    • waitUntil: (Optional) The conditions that must be met before the process is considered ready. See Wait conditions.
    • tty: (Optional) Run the process under a pseudo-terminal (PTY) instead of a pipe so full-screen terminal UIs render and accept keystrokes. Defaults to false. See Full-screen terminal UIs.
  • Output Saving: You can also save the command’s output using path, directory, maxVariation, and overwrite properties. See the runShell reference for details.

Output Regression Testing: You can save command output to a file and perform regression testing on subsequent runs. If path is specified, maxVariation is set, and a reference file exists at the target path, the action compares the new output to the reference file. If the difference is within maxVariation, the step passes. If it exceeds the threshold, the step returns WARNING, and behavior depends on the overwrite setting:

• If overwrite is set to "aboveVariation", the reference file is replaced with the new output, allowing you to capture updated artifacts for review.
• If overwrite is set to "never" (default), the reference file is not modified.
• If overwrite is set to "always", the reference file is always replaced regardless of variation.

This lets tests continue executing while flagging output changes that exceed your acceptable variation threshold.

Setting Variables: To capture output into variables for later steps, use the step-level variables object. You can assign values based on the command’s output using expressions like $$stdio.stdout, $$stdio.stderr, or $$exitCode. When you set stdio, the action also exposes $$stdioMatched, a boolean indicating whether the expected output was found in stdout or stderr. When you save output with path and a reference file already exists, the action exposes $$variation, the fractional difference between the new output and the reference file.

For comprehensive options, see the runShell reference.

Shell selection

Commands run in bash on every platform by default. POSIX syntax such as quoting, pipes, &&, and $() therefore works identically on Windows, macOS, and Linux.

• On macOS and Linux, Doc Detective uses the bash on your PATH.
• On Windows, Doc Detective uses Git Bash. It uses your Git for Windows install when you have one. Otherwise it downloads a portable copy (MinGit, about 40 MB) into its cache the first time a test needs it. You can pre-install it with doc-detective install bash. The bundled toolbox covers common utilities such as grep, sed, awk, and find. If your commands need more, install the full Git for Windows.

To run a command in a Windows shell instead, set shell to cmd or powershell on the step:

{
  "runShell": {
    "command": "echo %USERNAME%",
    "shell": "cmd"
  }
}

To change the default for a whole project, set shell in your config file:

{
  "shell": "powershell"
}

A step-level shell always wins over the config-level default. cmd and powershell steps fail on macOS and Linux. Gate them to Windows with runOn if your spec runs on multiple platforms. Background processes started with startSurface process descriptors use the config-level default too.

PowerShell notes: Windows PowerShell doesn’t support && chaining. Use ; there, or use the default bash. PowerShell also doesn’t propagate a native command’s exit code unless the command ends with exit $LASTEXITCODE. exitCodes assertions beyond [0, 1] are therefore unreliable under shell: "powershell". Prefer bash or cmd when asserting specific exit codes.

Background processes

To start a long-running process, such as a Docker container, development server, or database, add a background object. The step returns as soon as the process is ready instead of waiting for it to exit. The process keeps running while later steps and tests execute. To stop it, use a closeSurface step. To send keystrokes to an interactive process while it runs, use a type step with a surface. A startSurface step with a process descriptor starts the same kind of background process through the same launcher. Use whichever reads better in your test. startSurface also opens several surfaces concurrently in one step.

When you add a background object:

• name is required inside background. It identifies the process within the run so that a later closeSurface step can stop it, or a type step can target it. Doc Detective trims leading and trailing whitespace from the name, and so do the closeSurface and type steps, so the names match after trimming.
• timeout is the maximum time in milliseconds to wait for the process to become ready, not the maximum time the process can run.
• Doc Detective doesn’t apply the exitCodes, stdio, or output-saving (path, directory, maxVariation, overwrite) assertions, because the process doesn’t exit during the step.

The run owns the process. If you don’t stop it with a closeSurface step, Doc Detective stops it automatically when the run finishes, including when you interrupt the run with Ctrl+C.

A common pattern is to start the process in a setup test, run your tests against it, and stop it in a cleanup test. See the before and after test properties for setup and cleanup specs.

Wait conditions

Inside the background object, use waitUntil to block the step until the process is ready. If you omit waitUntil, Doc Detective treats the process as ready as soon as it starts. You can set any combination of the following conditions, and every condition you set must pass before timeout elapses.

• port: Wait until this TCP port accepts connections on localhost. An integer between 1 and 65535.
• stdio: Wait until the process output contains this content. Doc Detective searches both stdout and stderr and supports strings and regular expressions. To use a regular expression, start and end the string with a forward slash, as in /ready to accept/.
• httpGet: Wait until an HTTP GET request to this URL returns a 2xx status.
• delayMs: Wait at least this many milliseconds before considering the process ready. An integer, minimum 0.

If the process exits or doesn’t become ready before timeout elapses, the step fails and Doc Detective stops the process.

Some processes, such as certain Docker images and databases, fork a daemon and then exit. Because the foreground process doesn’t stay alive, Doc Detective treats it as having exited before it became ready, and the step fails. For these processes, use a port, httpGet, or delayMs condition instead of stdio.

Full-screen terminal UIs

By default, a background process runs behind a pipe. That works for line-oriented REPLs and basic CLIs, but not for full-screen terminal UIs (TUIs). Programs built with frameworks like Ink check whether their output is attached to a terminal (process.stdout.isTTY). They won’t render or accept keystrokes over a pipe. Set tty to true to run the process under a real pseudo-terminal (PTY) instead. These interactive TUIs then render and respond to the keystrokes a type step sends.

In PTY mode, the process’s stdout and stderr merge into a single stream that both waitUntil.stdio here and a type step’s waitUntil.stdio search. PTY output also includes raw ANSI escape sequences, such as colors and cursor movement. Match against text that renders without interleaved control codes, or use a regular expression that tolerates them. tty defaults to false, which keeps the standard pipe behavior.

Doc Detective guards the tty start against environment problems that could otherwise hang a run. Before each tty process starts, it verifies the pseudo-terminal backend’s files are intact on disk. It reinstalls them if a dependency install removed them mid-run. On Windows it also probes that a new pseudo-terminal can be allocated. If the backend can’t be restored or allocation is wedged, the step is skipped with an explanatory message instead of freezing the run.

The Windows allocation probe waits up to 15 seconds by default. To change that budget, set the DOC_DETECTIVE_PTY_PROBE_TIMEOUT_MS environment variable to a duration in milliseconds. Non-Windows platforms don’t run the probe.

Examples

Here are a few ways you might use the runShell action:

Run a basic command (string shorthand)

This example prints “hello world” to the output.

{
  "tests": [
    {
      "steps": [
        {
          "description": "Run a basic command.",
          "runShell": "echo hello world"
        }
      ]
    }
  ]
}

Because runShell uses bash on every platform by default, POSIX quoting works the same everywhere. echo 'hello world' prints without the quotes on Windows too. Only steps that opt into shell: "cmd" or shell: "powershell" need Windows-specific syntax.

Run a command with arguments (object format)

{
  "tests": [
    {
      "steps": [
        {
          "description": "Run echo with arguments.",
          "runShell": {
            "command": "echo",
            "args": ["hello", "world"]
          }
        }
      ]
    }
  ]
}

Run a command with expected output validation

This example runs a Docker container and checks the output for a specific string.

{
  "tests": [
    {
      "steps": [
        {
          "description": "Run a Docker container and check the output.",
          "runShell": {
            "command": "docker run hello-world",
            "stdio": "Hello from Docker!"
          }
        }
      ]
    }
  ]
}

Test a failure condition using exit codes

This example runs a command that exits with code 1 and checks for that exit code. Because exitCodes lists code 1, the step passes.

{
  "tests": [
    {
      "steps": [
        {
          "description": "Run a failing command and expect exit code 1.",
          "runShell": {
            "command": "exit 1",
            "exitCodes": [1]
          }
        }
      ]
    }
  ]
}

Set a variable based on command output

The first step echoes “setup”, validates the output using a regex, and captures the full stdout into a variable named TEST. The second step echoes the content of the TEST variable and validates that the output is indeed “setup”.

{
  "tests": [
    {
      "steps": [
        {
          "description": "Set a variable based on command output.",
          "runShell": {
            "command": "echo setup",
            "stdio": "/.+/"
          },
          "variables": {
            "TEST": "$$stdio.stdout"
          }
        },
        {
          "description": "Echo and validate the variable.",
          "runShell": {
            "command": "echo $TEST",
            "stdio": "setup"
          }
        }
      ]
    }
  ]
}

Start a background process and stop it later

This example starts an nginx container in the background, waits until http://localhost:8080 returns a 200 status code, and then stops the container with a closeSurface step.

{
  "tests": [
    {
      "steps": [
        {
          "description": "Start nginx in the background.",
          "runShell": {
            "command": "docker run -p 8080:80 nginx",
            "background": {
              "name": "web",
              "waitUntil": {
                "httpGet": "http://localhost:8080"
              }
            },
            "timeout": 30000
          }
        },
        {
          "description": "Stop the nginx container.",
          "closeSurface": "web"
        }
      ]
    }
  ]
}

Wait for several conditions at once

This example starts a Postgres container in the background. It waits until the database port accepts connections and the readiness message appears in the output.

{
  "tests": [
    {
      "steps": [
        {
          "description": "Start Postgres in the background.",
          "runShell": {
            "command": "docker run -p 5432:5432 -e POSTGRES_PASSWORD=test postgres:16",
            "background": {
              "name": "db",
              "waitUntil": {
                "port": 5432,
                "stdio": "/database system is ready to accept connections/"
              }
            },
            "timeout": 60000
          }
        },
        {
          "description": "Stop the database.",
          "closeSurface": "db"
        }
      ]
    }
  ]
}

Drive a full-screen TUI under a PTY

This example starts a full-screen terminal UI under a pseudo-terminal by setting tty to true. It waits for the prompt, drives the UI with type steps, and stops it with a closeSurface step. If @homebridge/node-pty-prebuilt-multiarch isn’t installed, the start step is skipped rather than failing.

{
  "tests": [
    {
      "steps": [
        {
          "description": "Start a TUI under a PTY and wait for its prompt.",
          "runShell": {
            "command": "my-tui",
            "background": {
              "name": "tui",
              "tty": true,
              "waitUntil": {
                "stdio": "/> $/"
              }
            },
            "timeout": 60000
          }
        },
        {
          "description": "Type a line and wait for the response.",
          "type": {
            "keys": ["hello", "$ENTER$"],
            "surface": "tui",
            "waitUntil": {
              "stdio": "/Hello/"
            }
          }
        },
        {
          "description": "Move the selection down and confirm.",
          "type": {
            "keys": ["$ARROW_DOWN$", "$ENTER$"],
            "surface": "tui"
          }
        },
        {
          "description": "Close the TUI.",
          "closeSurface": "tui"
        }
      ]
    }
  ]
}