Hand off test failures to an AI agent

When a test fails, Doc Detective’s GitHub Action can open an issue with the results and mention an AI agent in it. The agent analyzes the failure and suggests a fix. You review the suggestion and decide what to change.

How the handoff works

The handoff connects three components:

1. Automated testing: Doc Detective runs on every push or pull request, catching documentation that no longer matches your product.
2. Failure detection: When tests fail, the GitHub Action creates an issue with structured test results.
3. AI diagnosis: The issue mentions an AI integration that analyzes the failure and suggests fixes.

A person reviews every change before it lands. When documentation drifts from your product, the issue surfaces the problem with the agent’s analysis, and you decide on the fix. Tests can also update screenshots and command outputs. With create_pr_on_change, the action opens a pull request with those updates, and you review it before merging.

Set up the handoff

Configure your workflow to create issues with AI integration when tests fail:

name: doc-detective
on:
  push:
    branches: [main]
jobs:
  runTests:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      issues: write
    steps:
      - uses: actions/checkout@v4
      - uses: doc-detective/github-action@v1
        with:
          input: ./docs
          create_issue_on_fail: true
          token: ${{ secrets.DOC_DETECTIVE_ISSUE_TOKEN }}
          integrations: claude
          prompt: |
            Analyze these Doc Detective test failures. For each failure:
            1. Identify the root cause (documentation drift, broken link, UI change, etc.)
            2. Suggest a specific fix with the exact text or code changes needed
            3. Flag if the issue requires product changes vs documentation changes

This configuration:

• Runs tests against the ./docs directory on every push to main
• Creates a GitHub issue when any test fails, using the token in the DOC_DETECTIVE_ISSUE_TOKEN secret
• Mentions Claude in the issue with a detailed diagnostic prompt
• Passes the test results to Claude for analysis

Claude Code answers from a GitHub Actions workflow that starts when an issue opens. GitHub doesn’t start workflows for events that the default GITHUB_TOKEN creates. An issue the action opens with that token therefore never reaches Claude. Set token to one of these:

• A fine-grained personal access token that can write issues, stored as a repository secret. Its owner needs write access to the repository, because Claude Code ignores issues from users without it.
• A GitHub App token that the workflow generates on each run, for example with actions/create-github-app-token. Installation tokens expire after an hour, so don’t store one as a secret. Claude Code ignores bots by default, so add the app’s bot to allowed_bots in the Claude Code workflow.

The action also opens pull requests with token. If you set create_pr_on_change, as the weekly example does, the token also needs Pull requests: write.

The example runs on pushes to main, not on pull_request. By default, a pull request from a fork gets no secrets and a read-only GITHUB_TOKEN, so the action can’t open the issue there. A private repository can change that under Fork pull request workflows, as Tests are code explains. To check pull requests, keep a separate pull_request workflow without create_issue_on_fail.

Craft effective prompts

The prompt input determines how AI integrations analyze failures. A good prompt produces actionable suggestions rather than generic advice.

Prompt structure

Build your prompt around three elements:

Example prompts

For API documentation:

prompt: |
  These tests validate API documentation. For each failure:
  - Check if the API endpoint, request format, or response changed
  - Provide the corrected curl command or code example
  - Note if this suggests an undocumented API change

For procedural documentation:

prompt: |
  These tests validate step-by-step procedures. For each failure:
  - Identify which step broke and why
  - Check if the UI, command output, or workflow changed
  - Suggest updated instructions with exact wording

For screenshot tests:

prompt: |
  These visual regression tests compare screenshots against baselines. For each failure:
  - Describe what changed visually
  - Assess if this is a cosmetic change, feature change, or bug
  - Recommend whether to update the baseline or report a product issue

Prompt variables

Use these variables in your prompt or issue body:

Choose an AI integration

Each integration behaves differently. Choose based on your workflow:

Combine integrations to get multiple perspectives:

integrations: claude,copilot

Advanced configuration

Custom issue templates

Override the default issue body to structure how results appear:

issue_body: |
  ## Test Failure Report
  
  **Workflow run:** $RUN_URL
  
  ### Test Results
  $RESULTS
  
  ### Analysis Request
  $PROMPT

Scheduled runs

Run tests on a schedule to catch drift before it reaches production:

on:
  schedule:
    - cron: '0 6 * * *'  # Daily at 6 AM UTC
  push:
    branches: [main]

Auto-assign for triage

Route issues to specific team members:

issue_assignees: docs-team-lead,qa-engineer
issue_labels: doc-detective,needs-triage,automated

Example: weekly docs health check

This workflow combines scheduled runs, AI diagnosis, and automatic PR creation:

name: doc-detective-health-check
on:
  schedule:
    - cron: '0 6 * * 1'  # Weekly on Monday
  workflow_dispatch:  # Allow manual triggers
jobs:
  test-and-triage:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
      issues: write
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: doc-detective/github-action@v1
        with:
          input: ./docs
          create_issue_on_fail: true
          create_pr_on_change: true
          token: ${{ secrets.DOC_DETECTIVE_ISSUE_TOKEN }}
          integrations: claude,promptless
          prompt: |
            Analyze these documentation test failures and provide:
            1. Root cause for each failure
            2. Specific text changes to fix the documentation
            3. Whether any failures indicate product bugs vs doc bugs
            
            Format your response as a checklist the docs team can work through.
          
          issue_title: "Weekly docs health check: failures detected"
          issue_labels: doc-detective,automated,weekly-check
          
          pr_title: "Doc Detective: Updated screenshots and outputs"
          pr_labels: doc-detective,automated,for-review

This workflow:

• Runs weekly to catch gradual documentation drift
• Creates issues with Claude and Promptless analysis when tests fail
• Automatically opens PRs when tests update screenshots or command outputs
• Labels everything for straightforward filtering and triage