Tips & Gotchas for Proofarc UI Tests

A living list of lessons learned writing real tests against Proofarc's own UI. These are the patterns that look obvious in retrospect but cost a real debugging session the first time you hit them. Add to this page whenever you discover a new one.

Cross-engine: validation racing React render

Symptom. A test passes on WebDriver but fails on Playwright at a VALIDATE_TEXT body step, with the error showing only the static page chrome (sidebar / header text) and not the data the test expected.

Cause. Playwright is faster than WebDriver, so it asserts on body before React has rendered the page's async data. WebDriver's slower synchronous waits happen to mask the timing gap.

Fix. Insert a WAIT_FOR_TEXT for a marker the page renders only after the data loads, immediately before the VALIDATE_TEXT:

- action: NAVIGATE_TO
  value: "{{baseUrl}}/ui-test-suites"
- action: WAIT_FOR_ELEMENT
  selector: "body"
  value: "10"
- action: VALIDATE_URL
  expectedText: "ui-test-suites"
# This wait is the fix — gives React time to render the suites list.
- action: WAIT_FOR_TEXT
  selector: "body"
  expectedText: "Regression"
  timeout: "10"
- action: VALIDATE_TEXT
  selector: "body"
  expectedText: "Regression"

If the page never renders the text, the test still fails — but for a meaningful reason (the data is missing) rather than a timing race.

Cross-engine: native .click() on a child element

Symptom. An EXECUTE_JS step that does document.querySelector(...).click() works on WebDriver but doesn't navigate or update state on Playwright.

Cause. When you click on an inner <span> (e.g. the text label inside a Material UI <ListItemButton>), the native click event fires on the span and is supposed to bubble up to the button's React onClick handler. On Playwright the propagation through React's delegated event listener is less reliable than on WebDriver — sometimes the handler doesn't fire.

Fix. Walk up to the closest interactive ancestor (role="button", button, or a) and click that instead:

- action: EXECUTE_JS
  value: |
    const spans = Array.from(document.querySelectorAll('.MuiListItemText-root span'));
    const target = spans.find(function(el) {
      return el.textContent.trim() === 'Presets' && el.offsetParent !== null;
    });
    if (target) {
      const clickable = target.closest('[role="button"], button, a') || target.parentElement;
      clickable.click();
    }

The el.offsetParent !== null check ensures we only click an element that's actually visible — sidebar items inside a collapsed <SubMenu> are in the DOM but display: none-equivalent, and clicking them is a no-op.

Sidebar two-level structure

Symptom. A test tries to click 'API Tests' / 'UI Tests' / 'Performance' / 'Test Suites' in the sidebar via document.querySelectorAll('.MuiListItemText-root span').forEach(el => ...) and the click is silently a no-op — the test ends up still on Dashboard.

Cause. These items live inside a collapsible <SubMenu> category ("API Testing", "UI Testing", "Performance", etc.) which Material UI mounts with unmountOnExit — when the category is collapsed, the inner items are not in the DOM at all.

Fix. Two options:

1. Direct navigation (simpler, less coverage of the sidebar UX):

   - action: NAVIGATE_TO
     value: "{{baseUrl}}/scenarios"

2. Two-step EXECUTE_JS (preserves the sidebar-click flow): first click the category header to expand it, then click the inner item. Patent-supportive — keeps EXECUTE_JS as the interaction primitive.

Top-level items like 'Dashboard', 'Projects', and items under a <SectionHeader> (e.g. 'Presets', 'Templates', 'Environments' under Configuration) are always rendered, so a single click works for those.

YAML multi-line code with value: |

Symptom. Embedding multi-line JavaScript inside a quoted YAML string breaks the parser:

- action: EXECUTE_JS
  value: "const x = 1;
          fetch(...);"          # ← YAML parse error

Fix. Use YAML's literal block scalar (| after the field name). Indentation under the | is preserved verbatim; no quote escaping needed:

- action: EXECUTE_JS
  value: |
    const x = 1;
    return fetch('/api/whatever').then(function(r) { return r.json(); });

The | syntax also works for any other multi-line field (e.g. long expectedText, multi-line comments).

Awaiting async work in EXECUTE_JS

Symptom. An EXECUTE_JS step does await fetch(...), the call succeeds, but a follow-up VALIDATE_TITLE/VALIDATE_TEXT doesn't see the change.

Cause (details). WebDriver's executeScript does not await Promises — the script returns the unresolved Promise and the test moves on. Playwright's page.evaluate does await it.

Fix.

• For Playwright-only tests: use .then() chains and return the final Promise; pin driver: PLAYWRIGHT.
• For cross-engine tests: write a marker to document.title from inside the .then(), then a later VALIDATE_TITLE step asserts on it. The marker is observable from both engines.

- action: EXECUTE_JS
  value: |
    return fetch('/api/findings/60/status', {method: 'PUT', ...})
      .then(function(r) { return r.json(); })
      .then(function(body) {
        document.title = 'lifecycle-ok-' + body.status;
      });
- action: VALIDATE_TITLE
  expectedText: "lifecycle-ok"

Credentials in placeholders need explicit resolution

Symptom. Login fails; the screenshot shows the username field literally containing the text {{username}} and a "Login failed" banner.

Cause. The YAML uses {{username}} / {{password}} placeholders, but the run request didn't tell the backend which credential to use, and the environment has no credential marked isDefault: true. The agent sent the literal placeholder string to the form.

Fix. Either:

• Pass credentialTag: "admin" (or whichever tag) in the POST /api/ui-test-suites/{id}/run body.
• Mark one credential isDefault: true on the environment so the backend resolves it automatically.

See Credential Injection for the full resolution chain and the validation guard that catches this before dispatch.

Strict-mode selectors on Playwright

Symptom. A test fails on Playwright with strict mode violation: locator(...) resolved to N elements but the same selector works fine on WebDriver.

Cause. Playwright is strict by default — if a selector matches more than one element, locator calls throw. Selenium's findElement silently returns the first match.

Fix. Narrow the selector to a single element, or — if "any matching one" is genuinely the intent — use .first (the playwright-agent's VALIDATE_VISIBLE handler already does this internally so explicit .first in your YAML isn't usually needed).

# Bad: ".MuiDrawer-root, nav" matches 3 elements
- action: VALIDATE_VISIBLE
  selector: ".MuiDrawer-root, nav"

# Better: pick one
- action: VALIDATE_VISIBLE
  selector: "nav[aria-label='navigation menu']"

Cleanup data so tests are idempotent

Symptom. A test passes on first run, fails on second run.

Cause. The test created data (e.g. a scenario, a finding state) and left it dangling. The next run sees the leftover and behaves differently.

Fix. Every test should either:

• Use a read-only fixture (e.g. seeded by SQL once, never modified during the test), or
• Restore state in its final steps (e.g. transition a finding back to OPEN, or DELETE the scenario it created).

The Patent #2 lifecycle test is the canonical example: it ends by transitioning the seeded finding back to OPEN, so re-running produces the same result.

Engine recommendation is a hint, not a guarantee

Symptom. The wizard recommends HtmlUnit for a target that turns out to be a React SPA behind a login page — the crawl returns very few elements.

Cause. The recommender probe is unauthenticated. It fetches the public landing page once and classifies based on what it sees. If the landing page is a static login form (server-rendered HTML) but the real app behind the login is React, the probe can't see the SPA shape.

Fix. Link the target to a ProjectApplication and set its application_tech field (e.g. REACT). The recommender checks the DB first — that lookup beats the probe, so the auth-walled SPA gets the correct engine before any HTTP call.

If you can't link the target, just click the Playwright toggle on Step 2 — your override is sticky for the rest of the wizard session.

Don't trust LOW-confidence recommendations

The recommender returns confidence: LOW when no signal was strong enough. The wizard intentionally doesn't show a "Recommended" chip for LOW results — Playwright is already the safe default in that case. If you build automation around the API directly (GET /api/crawl/recommend-engine), filter confidence === 'LOW' out before acting on it, otherwise you'll second-guess a non-decision.

What a failed step gives you

Every failed step carries a picture of the screen at the moment it broke, on both engines:

{
  "action": "WAIT_FOR_ELEMENT",
  "status": "FAILED",
  "duration": 5.857,
  "errorMessage": "Expected condition failed: waiting for presence of element located by:
                   By.cssSelector: #results (tried for 5 second(s) with 500 milliseconds interval)",
  "screenshotUrl": "/api/screenshots/<jobId>/<scanId>/step1-368ea397.png"
}

That picture is the fastest way to tell apart the two explanations that produce an identical error message: the page never loaded and the page loaded and the selector is wrong. Open it first, before rewriting anything.

Fetching it needs a session (changed 2026-08-18)

/api/screenshots/... used to be readable by anyone who had the link. It is not any more — a run screenshot is a full-size picture of a logged-in screen in your own application, and the URL travels into browser history, Referer headers, proxy logs and anywhere it is pasted.

In the UI nothing changes; it fetches the image for you. From a script, send your token:

curl -H "Authorization: Bearer $TOKEN" "https://app.proofarc.ai/api/screenshots/<jobId>/<scanId>/<name>.png"

Without it you get 401.

Playwright attaches it too (changed 2026-08-18)

The screenshot was always captured on Playwright, but it went into a separate run-level list that the step result never pointed at — so a failed Playwright step showed no picture, while WebDriver showed one. Both engines now attach it to the step that failed.

A navigation that lands on an error page fails there

NAVIGATE_TO records the URL it asked for, the URL it ended up on, and the HTTP status. A 4xx or 5xx fails that step, naming the URL and the code:

Navigation to https://demo.example.com/app/login/app/login returned HTTP 404. The page
loaded, but it is not the page the test asked for, so any wait after this step would
time out against the wrong page.

Why this is worth knowing about. An error page loads in a fraction of a second and returns perfectly good HTML, so before this the navigation step passed — and the test then sat for its full timeout waiting for an element that was never going to be there. What you saw was timed out waiting for input[name=username], which reads as a slow or broken application. It was a bad URL.

A redirect is not an error: by the time the browser reports, the redirect has been followed, so the status is the page you actually landed on.

The status comes from the browser's own navigation timing, which Chrome and Firefox both report — verified on Chromium 145 and Firefox 131 against a real 200 and a real 404. If some future browser does not expose it, the step is left alone rather than failed: a check we cannot make is not a reason to fail your test.

A failure also stops the rest of the run, so a TAKE_SCREENSHOT step placed after the risky one never executes. Don't rely on one for failure evidence — the automatic capture is there precisely because the manual one cannot fire when you need it.

Screenshot links are not access-controlled today

Anyone with the URL can fetch the image, without a token. Treat a screenshot link like the screen it shows: if the run was against an app with real data on it, don't paste the link anywhere you would not paste the screenshot itself.