Skip to main content

Crawler & Discovery

The crawler discovers pages and elements on a target site so the wizard (and you) can build tests against real selectors instead of guessing. This page explains how the engines differ, when to pick each, how authenticated crawling works, and the full API surface — everything testers need to operate the crawler without reading the source.

Three engines, three jobs​

EngineTool name (internal)JavaScriptAuth supportSPA route discoverySpeedBest for
PlaywrightPLAYWRIGHT_CRAWLERFull ChromiumYes (form-based)YesSlowestReact, Angular, Vue, Next.js, anything client-rendered
HtmlUnitHTMLUNIT_CRAWLERLimited (Rhino JS)Yes (form-based)Configurable patternsMediumServer-rendered apps with light JS, Spring MVC, Rails, Django, JSP
SimpleWEB_CRAWLERNoneNoNoFastestPure HTML sites, sitemap-style traversal, robots discovery

Recommendation: let the wizard pick. Engine selection is automatic — the wizard calls GET /api/crawl/recommend-engine when you choose a target and pre-selects the best fit. A green Recommended chip shows the reason. You can always override with the toggle. See Creating Tests § Engine recommendation for the full signal table.

When to override the recommendation​

The recommender is good but not omniscient. Switch engines manually when:

  • The probe sees a login wall, the real app is a SPA. The recommender does an unauthenticated GET and can't see what's behind the login. If you know the app is React/Vue/Angular, click Playwright. Better still: link the target to a ProjectApplication with application_tech set, and the DB lookup wins over the probe.
  • You want screenshots. Only Playwright captures screenshots reliably during the crawl.
  • You're hitting an API gateway in front of a static site. The gateway's response can confuse the SPA-marker scan. Pick HtmlUnit explicitly.
  • You're crawling a sitemap. The Simple engine is fastest if you don't need element discovery — just URL enumeration.

Authenticated crawling​

Many real apps don't return anything useful until you log in. The crawler supports form-based authentication out of the box.

Wizard flow​

  1. In wizard Step 1, select a target that has credentials configured on its environment.
  2. In Step 2, pick a credential tag from the Login Credentials dropdown. (Or leave it blank to use the environment default.)
  3. The crawler:
    • Resolves the credential tag through EnvironmentAuthService (server-side — credentials never leave the backend in plaintext).
    • Merges in the target's stored login selectors from environment_target.auth_config (usernameField, passwordField, submitButton, loginUrl).
    • Falls back to generic CSS patterns + data-testid lookups if no selectors are stored.

Direct API flow​

POST /api/crawl accepts an auth block OR a credentialTag + environmentId pair. Either works:

Inline credentials (only use for ad-hoc testing — see Credentials guide):

{
  "target": "{{baseUrl}}",
  "engine": "PLAYWRIGHT",
  "auth": {
    "loginUrl": "/login",
    "usernameField": "#username",
    "passwordField": "#password",
    "submitButton": "button[type=submit]",
    "username": "{{username}}",
    "password": "{{password}}"
  }
}

Server-resolved credentials (preferred):

{
  "target": "{{baseUrl}}",
  "engine": "PLAYWRIGHT",
  "credentialTag": "admin",
  "environmentId": 1,
  "targetId": 42
}

The targetId field tells the backend to merge the target's stored login selectors. Without it, the agent uses generic patterns that may miss modern SPAs.

SPA route discovery​

SPAs don't have real URLs for every page — they have a single shell that mounts different views via client-side routing. The crawler handles this two ways.

Playwright (automatic)​

Playwright's crawler watches for pushState / replaceState events and treats each new path as a discovered page. No configuration needed.

HtmlUnit (configurable patterns)​

Pass urlExtractionPatterns and urlDataAttributes to POST /api/crawl when the SPA uses non-standard navigation:

{
  "target": "{{baseUrl}}",
  "engine": "HTMLUNIT",
  "urlExtractionPatterns": [
    { "pattern": "data-navigate=\\\"([^\\\"]+)\\\"", "group": 1 }
  ],
  "urlDataAttributes": ["data-route", "data-href"]
}

The HtmlUnit agent applies the regex against rendered HTML and follows any matches. Attributes are checked on every element.

POST /api/crawl/suggest-patterns can guess these patterns for a given URL.

Reading a crawl in the UI​

A crawl run shows up as a job record at Scans → the crawl job. It renders a crawl-specific result panel, not the security-findings table:

  • a headline — "N elements discovered across M page(s)"
  • one row per crawled page: title, URL, element count
  • expand a page to see the elements themselves — semantic name, tag, primary selector, text

The element detail is fetched on demand from GET /api/crawl/{jobId}/elements, so opening the job stays cheap. A crawl is never reported as "0 findings" — it has no findings by design; its output is the inventory.

API surface​

All endpoints live under /api/crawl. JWT required (ADMIN or ANALYST for write, VIEWER allowed on reads).

Start a crawl​

POST /api/crawl

Request body (all fields):

FieldTypeNotes
targetstring (required)URL to crawl
enginestringPLAYWRIGHT (default in wizard), HTMLUNIT, or SIMPLE
maxDepthintDefault 2
maxUrlsintDefault 50
authobjectInline auth — see above
credentialTagstringTag to resolve via EnvironmentAuthService
environmentIdlongRequired when credentialTag is set
targetIdlongEnables stored-selector merge
urlExtractionPatternsarrayHtmlUnit only
urlDataAttributesarrayHtmlUnit only

Response: { jobId, target, engine, status }. Poll with GET /api/crawl/{jobId}/elements.

Crawl a configured target​

POST /api/crawl/target/{targetId}

Reads auth_config and any stored selectors from the target. Optional body: { maxDepth, maxUrls }. Engine is HtmlUnit (the legacy default for this endpoint — switch to POST /api/crawl with targetId if you want Playwright).

Get crawl results​

GET /api/crawl/{jobId}/elements

Returns:

{
  "jobId": "abc-123",
  "status": "COMPLETED",
  "pages": [
    {
      "url": "{{baseUrl}}/login",
      "title": "Sign in",
      "elements": [
        { "tag": "input", "type": "text", "id": "username", "cssSelector": "#username" },
        { "tag": "button", "type": "submit", "text": "Sign in", "cssSelector": "button[type=submit]" },
        { "tag": "a", "type": "link", "text": "Learn more", "href": "https://tally.so/r/EkAPMl", "cssSelector": "a.cta" }
      ],
      "forms": [...]
    }
  ]
}

Status values: PENDING, RUNNING, COMPLETED, FAILED. The wizard polls every 5 seconds and stops when the status flips terminal.

Grounding attribute assertions (release-repo #577). Each element carries its key attributes — href (link destination), aria-label, role, value — alongside id, name, type, and placeholder. The compact digest (?mode=digest) also includes href on links. Use these to ground a VALIDATE_ATTRIBUTE assertion on the exact captured value (e.g. a link's real href) instead of guessing — attribute assertions are exact matches, so a guessed value fails.

Latest cached crawl​

GET /api/crawl/target/{targetId}/latest
GET /api/crawl/latest?url={url}

Returns the most recent completed crawl from the last 24 hours — the full inventory, plus the scope that run was crawled at:

{
  "jobId": "abc-123",
  "status": "COMPLETED",
  "engine": "PLAYWRIGHT_CRAWLER",
  "ageHours": 2.5,
  "maxDepth": 2,
  "maxUrls": 50,
  "pages": [ { "url": "...", "title": "...", "elements": [ ... ] } ]
}

The by-target form returns 404 when nothing is cached; the by-URL form returns 204. Use the caps to decide whether the cached run answers what you are about to ask for — a crawl that ran at maxUrls: 2 does not answer a request for 50. Crawls from all three engines are searched.

Discover form fields on a single page​

POST /api/crawl/discover-fields

Body: { "url": "{{baseUrl}}/login" }. Returns the form's input fields and the most likely submit button. Used by the wizard's credential configuration step to suggest selectors when none are stored.

SPA blind spot

discover-fields fetches the URL as plain HTML — no JavaScript executes. If the login form is rendered by React / Vue / Angular (mount div empty in the initial HTML, fields injected at runtime), the response will come back with empty inputs[] / forms[] arrays even though a human sees the form fine in a browser. Same blind spot as Recommend Engine.

When this happens you have two options:

  • Use the Visual Element Picker (which uses a real headless browser) to point at the fields directly, or
  • Crawl the login page first with engine: PLAYWRIGHT, then read /elements to see the rendered DOM.

Audited 2026-05-14: confirmed empty response when probing https://app.proofarc.ai/login (a React SPA).

Suggest SPA URL extraction patterns​

POST /api/crawl/suggest-patterns

Body: { "url": "{{baseUrl}}" }. Inspects the page and returns candidate urlExtractionPatterns and urlDataAttributes you can pass to a subsequent HtmlUnit crawl.

Recommend engine​

GET /api/crawl/recommend-engine?targetId={id}
GET /api/crawl/recommend-engine?url={url}

Returns { engine, confidence, reason }. See the full spec in API Reference § Recommend Engine.

What the crawl tells you about your own app​

Every crawled element carries a grade for how well a test can refer to it, and the crawl returns a verdict on the whole surface — the same grades and bands on web, Android and iOS.

"testabilityAdvisory": {
  "score": 62, "band": "NEEDS_WORK",
  "controlsAssessed": 34, "wellNamed": 21, "needsAttention": 13,
  "summary": "21 of 34 controls have a stable identifier. 13 can only be found by text
              or position, so tests touching them will break when the layout changes."
}

Per element: humanName, nameQuality (GOOD/WEAK/MISSING), namingSuggestion, selectorType, and clickable / isInput / isPassword / enabled.

Full detail: UI Testability Advisory — how the grade is decided, how the score is calculated, and what it does not tell you.

Was the crawl deep enough?​

Every crawl comes back with a verdict on its own completeness, because a coverage number computed from a half-finished crawl does not just carry more uncertainty — it reads better. The screens the crawl never reached are absent from the denominator, so a thin crawl scores higher than a thorough one.

"saturation": {
  "verdict": "TRUNCATED",
  "mayBackCoverage": false,
  "pagesRead": 50,
  "distinctFound": 31,
  "dryStreakAtEnd": 0,
  "stopReason": "BUDGET_EXHAUSTED",
  "discoveryCurve": [
    {"afterPages": 5,  "newFound": 5},
    {"afterPages": 10, "newFound": 4},
    {"afterPages": 45, "newFound": 2},
    {"afterPages": 50, "newFound": 3}
  ],
  "summary": "The crawl stopped because it hit its page limit, not because it ran out of app.
              There is more surface it never reached, so a coverage number from this crawl would
              be flattering — raise maxUrls and run it again."
}
VerdictMeaning
SATURATEDThe last several pages found nothing new. The crawled surface looks exhausted
TRUNCATEDHit the page limit. There is more it never reached
NOT_SATURATEDStill finding new screens when it ended. Crawl deeper
INSUFFICIENT_DATAToo few pages to say either way

The discovery curve is the thing to look at. New screens per five pages. A curve that flattens and stays flat is the evidence the crawl exhausted the app. One still climbing when the crawl ended means there was more to find.

TRUNCATED is unconditional

A crawl that hit its page limit is never reported as saturated, however flat its curve looked. The limit is exactly what stopped it from finding out whether there was more, so a flat tail at the end proves nothing.

Raise maxUrls and run it again. pendingUrls tells you how many links it had already found and never visited.

Pages, screens and variants​

A crawl visits pages. You test screens, and they are not the same thing: /orders/1 and /orders/2 are one screen showing different records. Counting them separately would make a crawl of a busy database look enormous while covering a single template.

"screenSummary": {
  "screens": 22,
  "variants": 27,
  "pagesVisited": 340,
  "instancesCollapsed": 313,
  "detail": [
    {
      "screen": "/orders/{id}",
      "instances": 300,
      "variants": [
        {"signature": "3f9a2b1c4d5e", "controls": 12, "instances": 299,
         "exampleUrls": ["/orders/1", "/orders/2", "/orders/3"]},
        {"signature": "7c1e8d40aa93", "controls": 3, "instances": 1,
         "exampleUrls": ["/orders/8814"]}
      ]
    }
  ]
}

Screen is the route with record ids collapsed. Variant is a distinct shape within it — the second one above is the empty state, seen once, and almost certainly the one with no test.

instancesCollapsed is worth reading on its own: "340 pages visited, 22 screens" tells you immediately whether the crawl's budget went somewhere useful or was spent walking a table.

What splits a screen, and what does not

Record ids do not: /orders/1, /orders/8f3a-…-91b and ?page=2 are all one screen. Neither does content — two orders differ in every label and are the same screen.

A genuinely different shape does, and that is deliberate: an SPA /settings with four panels, a wizard that keeps one URL across steps, or a list in its empty state, are different things to test.

Result caching​

The backend caches completed crawls for 24 hours per target URL. A new crawl of the same URL within that window is served from cache only when the cached run covered at least the scope you are asking for — its maxDepth and maxUrls were both greater than or equal to yours:

{
  "jobId": "abc-123",
  "target": "{{baseUrl}}",
  "status": "COMPLETED",
  "cached": true,
  "completedAt": "2026-05-13T10:14:22Z",
  "maxDepth": 3,
  "maxUrls": 100
}

Widening the scope re-crawls. Ask for maxUrls: 50 after a cached maxUrls: 2 run and the crawler runs again — you get a new jobId and the wider page set. Narrowing does not: a cached wide crawl already contains everything a narrower request would find. A cached run with no recorded caps (rows predating this) is not provably wide enough, so it re-crawls.

The cached: true flag and the maxDepth/maxUrls on the response tell you which run you are looking at — reuse is never silent. Playwright crawls always run fresh.

This used to be wrong

Before 86bbaph8d, reuse ignored the caps entirely: a re-run with different maxUrls/maxDepth returned the earlier crawl's jobId, crawledAt and page set, so the scope knobs did nothing on any already-crawled target and authoring worked off an element set that did not match the request.

Common gotchas​

See Tips & Gotchas for the full list. Crawler-specific entries:

  • Engine recommendation is a hint, not a guarantee. The probe is unauthenticated — auth-walled SPAs may misclassify. Link the target to a ProjectApplication to skip the probe.
  • confidence: LOW is silent. The wizard shows no chip for LOW results — that's not a bug, it's "no signal worth surfacing, Playwright is already the safe default."
  • HtmlUnit on heavy SPAs returns near-empty results. If a Playwright crawl returns hundreds of elements but HtmlUnit returns three, the site needs Playwright. The smart-crawler suggestion in the wizard surfaces this as a callout: "Looks like a SPA — try Playwright or the Visual Picker."
  • Crawl jobs run on agent containers. proofarc-playwright runs Playwright crawls, proofarc-ui-test runs HtmlUnit crawls, proofarc-scanner runs Simple crawls. If a job stays PENDING indefinitely, check the relevant agent is healthy.