Tips & Gotchas for Proofarc UI Tests

A living list of lessons learned writing real tests against Proofarc's own UI. These are the patterns that look obvious in retrospect but cost a real debugging session the first time you hit them. Add to this page whenever you discover a new one.

Cross-engine: validation racing React render

Symptom. A test passes on WebDriver but fails on Playwright at a VALIDATE_TEXT body step, with the error showing only the static page chrome (sidebar / header text) and not the data the test expected.

Cause. Playwright is faster than WebDriver, so it asserts on body before React has rendered the page's async data. WebDriver's slower synchronous waits happen to mask the timing gap.

Fix. Insert a WAIT_FOR_TEXT for a marker the page renders only after the data loads, immediately before the VALIDATE_TEXT:

- action: NAVIGATE_TO
  value: "{{baseUrl}}/ui-test-suites"
- action: WAIT_FOR_ELEMENT
  selector: "body"
  value: "10"
- action: VALIDATE_URL
  expectedText: "ui-test-suites"
# This wait is the fix — gives React time to render the suites list.
- action: WAIT_FOR_TEXT
  selector: "body"
  expectedText: "Regression"
  timeout: "10"
- action: VALIDATE_TEXT
  selector: "body"
  expectedText: "Regression"

If the page never renders the text, the test still fails — but for a meaningful reason (the data is missing) rather than a timing race.

Cross-engine: native .click() on a child element

Symptom. An EXECUTE_JS step that does document.querySelector(...).click() works on WebDriver but doesn't navigate or update state on Playwright.

Cause. When you click on an inner <span> (e.g. the text label inside a Material UI <ListItemButton>), the native click event fires on the span and is supposed to bubble up to the button's React onClick handler. On Playwright the propagation through React's delegated event listener is less reliable than on WebDriver — sometimes the handler doesn't fire.

Fix. Walk up to the closest interactive ancestor (role="button", button, or a) and click that instead:

- action: EXECUTE_JS
  value: |
    const spans = Array.from(document.querySelectorAll('.MuiListItemText-root span'));
    const target = spans.find(function(el) {
      return el.textContent.trim() === 'Presets' && el.offsetParent !== null;
    });
    if (target) {
      const clickable = target.closest('[role="button"], button, a') || target.parentElement;
      clickable.click();
    }

The el.offsetParent !== null check ensures we only click an element that's actually visible — sidebar items inside a collapsed <SubMenu> are in the DOM but display: none-equivalent, and clicking them is a no-op.

Sidebar two-level structure

Symptom. A test tries to click 'API Tests' / 'UI Tests' / 'Performance' / 'Test Suites' in the sidebar via document.querySelectorAll('.MuiListItemText-root span').forEach(el => ...) and the click is silently a no-op — the test ends up still on Dashboard.

Cause. These items live inside a collapsible <SubMenu> category ("API Testing", "UI Testing", "Performance", etc.) which Material UI mounts with unmountOnExit — when the category is collapsed, the inner items are not in the DOM at all.

Fix. Two options:

1. Direct navigation (simpler, less coverage of the sidebar UX):

   - action: NAVIGATE_TO
     value: "{{baseUrl}}/scenarios"

2. Two-step EXECUTE_JS (preserves the sidebar-click flow): first click the category header to expand it, then click the inner item. Patent-supportive — keeps EXECUTE_JS as the interaction primitive.

Top-level items like 'Dashboard', 'Projects', and items under a <SectionHeader> (e.g. 'Presets', 'Templates', 'Environments' under Configuration) are always rendered, so a single click works for those.

YAML multi-line code with value: |

Symptom. Embedding multi-line JavaScript inside a quoted YAML string breaks the parser:

- action: EXECUTE_JS
  value: "const x = 1;
          fetch(...);"          # ← YAML parse error

Fix. Use YAML's literal block scalar (| after the field name). Indentation under the | is preserved verbatim; no quote escaping needed:

- action: EXECUTE_JS
  value: |
    const x = 1;
    return fetch('/api/whatever').then(function(r) { return r.json(); });

The | syntax also works for any other multi-line field (e.g. long expectedText, multi-line comments).

Awaiting async work in EXECUTE_JS

Symptom. An EXECUTE_JS step does await fetch(...), the call succeeds, but a follow-up VALIDATE_TITLE/VALIDATE_TEXT doesn't see the change.

Cause (details). WebDriver's executeScript does not await Promises — the script returns the unresolved Promise and the test moves on. Playwright's page.evaluate does await it.

Fix.

• For Playwright-only tests: use .then() chains and return the final Promise; pin driver: PLAYWRIGHT.
• For cross-engine tests: write a marker to document.title from inside the .then(), then a later VALIDATE_TITLE step asserts on it. The marker is observable from both engines.

- action: EXECUTE_JS
  value: |
    return fetch('/api/findings/60/status', {method: 'PUT', ...})
      .then(function(r) { return r.json(); })
      .then(function(body) {
        document.title = 'lifecycle-ok-' + body.status;
      });
- action: VALIDATE_TITLE
  expectedText: "lifecycle-ok"

Credentials in placeholders need explicit resolution

Symptom. Login fails; the screenshot shows the username field literally containing the text {{username}} and a "Login failed" banner.

Cause. The YAML uses {{username}} / {{password}} placeholders, but the run request didn't tell the backend which credential to use, and the environment has no credential marked isDefault: true. The agent sent the literal placeholder string to the form.

Fix. Either:

• Pass credentialTag: "admin" (or whichever tag) in the POST /api/ui-test-suites/{id}/run body.
• Mark one credential isDefault: true on the environment so the backend resolves it automatically.

See Credential Injection for the full resolution chain and the validation guard that catches this before dispatch.

Strict-mode selectors on Playwright

Symptom. A test fails on Playwright with strict mode violation: locator(...) resolved to N elements but the same selector works fine on WebDriver.

Cause. Playwright is strict by default — if a selector matches more than one element, locator calls throw. Selenium's findElement silently returns the first match.

Fix. Narrow the selector to a single element, or — if "any matching one" is genuinely the intent — use .first (the playwright-agent's VALIDATE_VISIBLE handler already does this internally so explicit .first in your YAML isn't usually needed).

# Bad: ".MuiDrawer-root, nav" matches 3 elements
- action: VALIDATE_VISIBLE
  selector: ".MuiDrawer-root, nav"

# Better: pick one
- action: VALIDATE_VISIBLE
  selector: "nav[aria-label='navigation menu']"

Cleanup data so tests are idempotent

Symptom. A test passes on first run, fails on second run.

Cause. The test created data (e.g. a scenario, a finding state) and left it dangling. The next run sees the leftover and behaves differently.

Fix. Every test should either:

• Use a read-only fixture (e.g. seeded by SQL once, never modified during the test), or
• Restore state in its final steps (e.g. transition a finding back to OPEN, or DELETE the scenario it created).

The Patent #2 lifecycle test is the canonical example: it ends by transitioning the seeded finding back to OPEN, so re-running produces the same result.

Engine recommendation is a hint, not a guarantee

Symptom. The wizard recommends HtmlUnit for a target that turns out to be a React SPA behind a login page — the crawl returns very few elements.

Cause. The recommender probe is unauthenticated. It fetches the public landing page once and classifies based on what it sees. If the landing page is a static login form (server-rendered HTML) but the real app behind the login is React, the probe can't see the SPA shape.

Fix. Link the target to a ProjectApplication and set its application_tech field (e.g. REACT). The recommender checks the DB first — that lookup beats the probe, so the auth-walled SPA gets the correct engine before any HTTP call.

If you can't link the target, just click the Playwright toggle on Step 2 — your override is sticky for the rest of the wizard session.

Don't trust LOW-confidence recommendations

The recommender returns confidence: LOW when no signal was strong enough. The wizard intentionally doesn't show a "Recommended" chip for LOW results — Playwright is already the safe default in that case. If you build automation around the API directly (GET /api/crawl/recommend-engine), filter confidence === 'LOW' out before acting on it, otherwise you'll second-guess a non-decision.