Visual Element Picker

The Visual Element Picker is a click-to-select helper in the UI Test wizard. You see a live screenshot of the target app, click on a button or input, and the platform turns that click into a test step with the right CSS selector / XPath baked in — so you don't have to inspect the page DOM yourself.

It lives at:

UI Tests → New Test → Wizard → Step 2 "Discover Page" → "Pick from Page"

What the picker actually does

When you click "Pick from Page," the backend:

1. Opens a real headless Chrome session via Selenium against your target's URL.
2. Logs into the target if the environment has credentials configured (it uses EnvironmentAuthService and the env's default credential).
3. Takes a screenshot at 1920×1080 viewport and shows it to you in the dialog.

When you click on a spot in that screenshot, the backend:

1. Maps your click pixel back to the browser viewport, accounting for the dialog's image scaling.
2. Asks the browser what elements are at that point — using document.elementsFromPoint(x, y) (plural).
3. Walks the resulting z-stack and returns the first interactive element it finds (see "smart routing" below). Falls back to the topmost element if none of them are interactive.
4. Returns its tag, id, name, type, text, className, placeholder, value, xpath, cssSelector, semanticName, and rect so the wizard can build a step.

Smart routing — why clicking on a button gives you the button

A naïve picker uses document.elementFromPoint(x, y), which returns the topmost element in the z-stack. On MUI / Bootstrap / styled-form pages, that's usually the wrapping <div> or floating <label> z-above the actual <input>/<button>. A direct click on the visible control then picked the wrapper instead, and you had to click around the target to land on it. This was QA-reported as proofarc/cloud-platform-test-collaboration#31 and fixed on 2026-05-12.

The fix uses document.elementsFromPoint (plural — returns the full stack) and prefers an interactive element:

• Native form controls: INPUT, BUTTON, SELECT, TEXTAREA, A
• ARIA roles: button, link, checkbox, radio, tab, menuitem, switch
• Has an onclick handler
• Has a data-testid attribute (test-author signal)
• Has an explicit tabindex (focusable)

If the resolved element is a <label htmlFor="x">, the picker returns the labelled control instead — clicking a label visually targets the input it labels.

When no interactive element is in the stack (paragraph text, decorative images), the topmost element is still returned, so non-form clicks behave normally.

Selector preference order

The wizard fills in the new step's selector from the picker response. The platform prefers, in order:

1. #id — fastest, most stable.
2. [name='...'] — second-most-stable.
3. tag:nth-of-type(n) > ... — full CSS path. Stable as long as the DOM structure doesn't change.
4. xpath — included on every response but rarely needed; use it when the CSS path doesn't disambiguate.

A semanticName is also returned (element.id slugified, then name, then aria-label, then truncated innerText). The wizard uses this as the default step name so your test list reads like prose.

Controls in the dialog

Control	What it does
Pick from Page	Open the picker modal. First call starts a Selenium session (~5-10s).
Navigate	Send the browser to a different URL. Useful for picking elements on pages reachable by login but not at the start URL.
Scroll	Scrolls the headless browser. Use when the element you want is below the viewport fold.
Refresh	Re-takes the screenshot. Useful if the app re-renders client-side after the initial screenshot.
Close	Ends the Selenium session.

Limits + things the picker doesn't see

• Max 3 concurrent picker sessions platform-wide. If you hit the limit, close another one or wait for inactivity timeout.
• 10-minute inactivity timeout — if you leave the dialog open and don't interact, the session is reaped and you'll need to reopen.
• Iframes: clicks inside an iframe resolve against the iframe's own DOM, not the parent's. The picker handles this transparently but selectors may need an iframe-specific prefix when used in a test.
• Shadow DOM: document.elementsFromPoint doesn't pierce shadow boundaries. Elements inside a closed shadow root won't be returned. Use a manual selector with >>> or ::shadow syntax in your test instead.
• Modal overlays / portals: most MUI modals render via a Portal outside the main DOM tree but with a high z-index. The picker handles these because elementsFromPoint returns the full stack — including the portal content — and the interactive-element preference picks the modal's button over the backdrop.
• Hover-only menus: the picker takes a static screenshot, so menus that only appear on hover won't be visible. Use EXECUTE_JS in the test to dispatch mouseover first, or just type the selector by hand from devtools.
• Animations: if you click during a CSS transform/transition, the resolved element may be wrong because the visual position doesn't match the layout position. Wait for the animation to finish before clicking.

When to skip the picker and type the selector

The picker is great for: visible buttons, inputs, links, and most form elements on a typical SPA.

It struggles with: shadow-DOM web components, hover-only flyouts, very small targets (under 8 pixels), and elements behind animation. For those, open the browser's devtools, find a stable id or data-testid, and paste it into the step manually.

API endpoints (if you're calling it yourself)

Method	Path	Body
POST	/api/element-picker/start	{"url": "...", "authConfig": {...}}
POST	/api/element-picker/pick	{"sessionId":"...","x":100,"y":200,"displayWidth":1920,"displayHeight":1080}
POST	/api/element-picker/screenshot	{"sessionId": "..."}
POST	/api/element-picker/navigate	{"sessionId": "...", "url": "..."}
POST	/api/element-picker/scroll	{"sessionId": "...", "deltaY": 500}
DELETE	/api/element-picker/{sessionId}	(no body)

All require an authenticated user with the ADMIN or ANALYST role.

Reporting picker bugs

If the picker selects the wrong element, file an issue at proofarc/cloud-platform-test-collaboration/issues with:

• The target URL (so engineering can open the same page).
• A screenshot showing where you clicked and what was selected vs. what you expected.
• The resolved selector from the wizard — it's visible in the step row after you Add it.
• Whether the target uses MUI / Bootstrap / a custom design system (helps narrow the z-stack situation).