Crawler & Discovery

The crawler discovers pages and elements on a target site so the wizard (and you) can build tests against real selectors instead of guessing. This page explains how the engines differ, when to pick each, how authenticated crawling works, and the full API surface — everything testers need to operate the crawler without reading the source.

Three engines, three jobs

Engine	Tool name (internal)	JavaScript	Auth support	SPA route discovery	Speed	Best for
Playwright	`PLAYWRIGHT_CRAWLER`	Full Chromium	Yes (form-based)	Yes	Slowest	React, Angular, Vue, Next.js, anything client-rendered
HtmlUnit	`HTMLUNIT_CRAWLER`	Limited (Rhino JS)	Yes (form-based)	Configurable patterns	Medium	Server-rendered apps with light JS, Spring MVC, Rails, Django, JSP
Simple	`WEB_CRAWLER`	None	No	No	Fastest	Pure HTML sites, sitemap-style traversal, robots discovery

Recommendation: let the wizard pick. Engine selection is automatic — the wizard calls GET /api/crawl/recommend-engine when you choose a target and pre-selects the best fit. A green Recommended chip shows the reason. You can always override with the toggle. See Creating Tests § Engine recommendation for the full signal table.

When to override the recommendation

The recommender is good but not omniscient. Switch engines manually when:

The probe sees a login wall, the real app is a SPA. The recommender does an unauthenticated GET and can't see what's behind the login. If you know the app is React/Vue/Angular, click Playwright. Better still: link the target to a ProjectApplication with application_tech set, and the DB lookup wins over the probe.
You want screenshots. Only Playwright captures screenshots reliably during the crawl.
You're hitting an API gateway in front of a static site. The gateway's response can confuse the SPA-marker scan. Pick HtmlUnit explicitly.
You're crawling a sitemap. The Simple engine is fastest if you don't need element discovery — just URL enumeration.

Authenticated crawling

Many real apps don't return anything useful until you log in. The crawler supports form-based authentication out of the box.

Wizard flow

In wizard Step 1, select a target that has credentials configured on its environment.
In Step 2, pick a credential tag from the Login Credentials dropdown. (Or leave it blank to use the environment default.)
The crawler:
- Resolves the credential tag through EnvironmentAuthService (server-side — credentials never leave the backend in plaintext).
- Merges in the target's stored login selectors from environment_target.auth_config (usernameField, passwordField, submitButton, loginUrl).
- Falls back to generic CSS patterns + data-testid lookups if no selectors are stored.

Direct API flow

POST /api/crawl accepts an auth block OR a credentialTag + environmentId pair. Either works:

Inline credentials (only use for ad-hoc testing — see Credentials guide):

{
  "target": "{{baseUrl}}",
  "engine": "PLAYWRIGHT",
  "auth": {
    "loginUrl": "/login",
    "usernameField": "#username",
    "passwordField": "#password",
    "submitButton": "button[type=submit]",
    "username": "{{username}}",
    "password": "{{password}}"
  }
}

Server-resolved credentials (preferred):

{
  "target": "{{baseUrl}}",
  "engine": "PLAYWRIGHT",
  "credentialTag": "admin",
  "environmentId": 1,
  "targetId": 42
}

The targetId field tells the backend to merge the target's stored login selectors. Without it, the agent uses generic patterns that may miss modern SPAs.

SPA route discovery

SPAs don't have real URLs for every page — they have a single shell that mounts different views via client-side routing. The crawler handles this two ways.

Playwright (automatic)

Playwright's crawler watches for pushState / replaceState events and treats each new path as a discovered page. No configuration needed.

HtmlUnit (configurable patterns)

Pass urlExtractionPatterns and urlDataAttributes to POST /api/crawl when the SPA uses non-standard navigation:

{
  "target": "{{baseUrl}}",
  "engine": "HTMLUNIT",
  "urlExtractionPatterns": [
    { "pattern": "data-navigate=\\\"([^\\\"]+)\\\"", "group": 1 }
  ],
  "urlDataAttributes": ["data-route", "data-href"]
}

The HtmlUnit agent applies the regex against rendered HTML and follows any matches. Attributes are checked on every element.

POST /api/crawl/suggest-patterns can guess these patterns for a given URL.

API surface

All endpoints live under /api/crawl. JWT required (ADMIN or ANALYST for write, VIEWER allowed on reads).

Start a crawl

POST /api/crawl

Request body (all fields):

Field	Type	Notes
`target`	string (required)	URL to crawl
`engine`	string	`PLAYWRIGHT` (default in wizard), `HTMLUNIT`, or `SIMPLE`
`maxDepth`	int	Default `2`
`maxUrls`	int	Default `50`
`auth`	object	Inline auth — see above
`credentialTag`	string	Tag to resolve via `EnvironmentAuthService`
`environmentId`	long	Required when `credentialTag` is set
`targetId`	long	Enables stored-selector merge
`urlExtractionPatterns`	array	HtmlUnit only
`urlDataAttributes`	array	HtmlUnit only

Response: { jobId, target, engine, status }. Poll with GET /api/crawl/{jobId}/elements.

Crawl a configured target

POST /api/crawl/target/{targetId}

Reads auth_config and any stored selectors from the target. Optional body: { maxDepth, maxUrls }. Engine is HtmlUnit (the legacy default for this endpoint — switch to POST /api/crawl with targetId if you want Playwright).

Get crawl results

GET /api/crawl/{jobId}/elements

Returns:

{
  "jobId": "abc-123",
  "status": "COMPLETED",
  "pages": [
    {
      "url": "{{baseUrl}}/login",
      "title": "Sign in",
      "elements": [
        { "tag": "input", "type": "text", "id": "username", "cssSelector": "#username" },
        { "tag": "button", "type": "submit", "text": "Sign in", "cssSelector": "button[type=submit]" }
      ],
      "forms": [...]
    }
  ]
}

Status values: PENDING, RUNNING, COMPLETED, FAILED. The wizard polls every 5 seconds and stops when the status flips terminal.

Latest cached crawl for a target

GET /api/crawl/target/{targetId}/latest

Returns the most recent completed crawl from the last 24 hours, or 404 if none exists. The wizard uses this to short-circuit re-crawling.

Discover form fields on a single page

POST /api/crawl/discover-fields

Body: { "url": "{{baseUrl}}/login" }. Returns the form's input fields and the most likely submit button. Used by the wizard's credential configuration step to suggest selectors when none are stored.

SPA blind spot

discover-fields fetches the URL as plain HTML — no JavaScript executes. If the login form is rendered by React / Vue / Angular (mount div empty in the initial HTML, fields injected at runtime), the response will come back with empty inputs[] / forms[] arrays even though a human sees the form fine in a browser. Same blind spot as Recommend Engine.

When this happens you have two options:

Use the Visual Element Picker (which uses a real headless browser) to point at the fields directly, or
Crawl the login page first with engine: PLAYWRIGHT, then read /elements to see the rendered DOM.

Audited 2026-05-14: confirmed empty response when probing https://app.proofarc.ai/login (a React SPA).

Suggest SPA URL extraction patterns

POST /api/crawl/suggest-patterns

Body: { "url": "{{baseUrl}}" }. Inspects the page and returns candidate urlExtractionPatterns and urlDataAttributes you can pass to a subsequent HtmlUnit crawl.

GET /api/crawl/recommend-engine?targetId={id}
GET /api/crawl/recommend-engine?url={url}

Returns { engine, confidence, reason }. See the full spec in API Reference § Recommend Engine.

Result caching

The backend caches completed crawls for 24 hours per target. Starting a new crawl for the same target URL within that window returns the cached jobId instead of running again:

{
  "jobId": "abc-123",
  "target": "{{baseUrl}}",
  "status": "COMPLETED",
  "cached": true,
  "completedAt": "2026-05-13T10:14:22Z"
}

To force a fresh crawl, change the URL slightly (e.g. add a query string) or wait out the cache window.

Common gotchas

See Tips & Gotchas for the full list. Crawler-specific entries:

Engine recommendation is a hint, not a guarantee. The probe is unauthenticated — auth-walled SPAs may misclassify. Link the target to a ProjectApplication to skip the probe.
confidence: LOW is silent. The wizard shows no chip for LOW results — that's not a bug, it's "no signal worth surfacing, Playwright is already the safe default."
HtmlUnit on heavy SPAs returns near-empty results. If a Playwright crawl returns hundreds of elements but HtmlUnit returns three, the site needs Playwright. The smart-crawler suggestion in the wizard surfaces this as a callout: "Looks like a SPA — try Playwright or the Visual Picker."
Crawl jobs run on agent containers. proofarc-playwright runs Playwright crawls, proofarc-ui-test runs HtmlUnit crawls, proofarc-scanner runs Simple crawls. If a job stays PENDING indefinitely, check the relevant agent is healthy.

Creating Tests (UI) — wizard walkthrough including Step 2 (Discover)
Visual Element Picker — click-to-pick when crawling alone isn't enough
Credential Injection — how the crawler resolves credentials at runtime
Tips & Gotchas — debugging the crawler

Three engines, three jobs​

When to override the recommendation​

Authenticated crawling​

Wizard flow​

Direct API flow​

SPA route discovery​

Playwright (automatic)​

HtmlUnit (configurable patterns)​

API surface​

Start a crawl​

Crawl a configured target​

Get crawl results​

Latest cached crawl for a target​

Discover form fields on a single page​

Suggest SPA URL extraction patterns​

Recommend engine​

Result caching​

Common gotchas​

Related​