Skip to main content

UI testability advisory

Purpose: tell a team what makes their own application hard to automate, in terms a developer can act on — before anyone writes the tests that will break.

Every crawl grades each element on how well a test can refer to it, and returns a verdict on the whole surface. Web, Android and iOS use the same grades, the same score and the same bands, so an app can be compared with itself across platforms.

This is the one output of a crawl that improves the application rather than describing it.

Why it exists​

A test finds an element by something. If that something is a data-testid, the test survives a redesign. If it is the visible text, the test breaks on the next copy edit and does not work in another language. If there is nothing at all, the test can only use position on screen — and that breaks the first time anything moves.

Teams usually discover which of these they have by watching tests break for months. The crawl already knows on day one. It just never said so.

What you get​

Every element carries:

FieldMeaning
humanNameThe best readable name for it, or null when it has none
nameQualityGOOD · WEAK · MISSING
namingSuggestionWhat to change and why. null when nothing needs doing
selectorTypeWhich kind of locator the selector is — data-testid, accessibility, id, name, …
clickable isInput isPassword enabledWhat kind of control it is
platformWEB · ANDROID · IOS

And the crawl itself carries the verdict:

"testabilityAdvisory": {
  "score": 62,
  "band": "NEEDS_WORK",
  "controlsAssessed": 34,
  "wellNamed": 21,
  "needsAttention": 13,
  "summary": "21 of 34 controls have a stable identifier. 13 can only be found by text or
              position, so tests touching them will break when the layout or wording changes.",
  "examples": [ … up to 10 … ]
}

How an element is graded​

GOOD — it has an identifier a developer chose and meant to be addressed.

  • Web: data-testid, or aria-label
  • Android: content-desc, or a meaningful resource id
  • iOS: the accessibility identifier

A test written against it keeps working.

WEAK — addressable, but by something that will not last. Two cases:

  • Findable only by its visible text. Works today; breaks on the next wording change and fails in every other language.
  • An identifier that looks auto-generated — a3f9c2b1e4d5, view123, a UUID, untitled. This is the case worth understanding: it is addressable right now and different after the next build, which is worse than having none, because it looks fine.

MISSING — nothing to refer to at all. Only position or structure can find it.

The grading is deliberately conservative

Calling a good name bad trains people to ignore the report, which costs more than staying quiet about a borderline one. When in doubt the grade is the kinder one.

How the score is calculated​

score = well-named controls / interactive controls   (as a percentage)
BandScoreWhat it means
READY80–100Tests written today will survive UI changes
NEEDS_WORK50–79A meaningful share of controls will break tests when the layout moves
HARD_TO_AUTOMATE0–49Most controls can only be found by text or position
NO_CONTROLS—Nothing interactive was found, so there is nothing to assess

Only interactive controls are scored — buttons, inputs, links, selects, forms, and their mobile equivalents. Body text having no identifier does not make an application harder to automate, and a score that punished prose is one people stop reading.

A page of pure content returns band: "NO_CONTROLS" and no score, rather than 0. Zero out of zero is not a failing grade.

Reading the result​

The examples list is the actionable part. Each entry names a control, why it is a problem, and what to do:

{
  "type": "Button",
  "nameQuality": "WEAK",
  "text": "Continue",
  "selector": "button.btn-primary:nth-of-type(2)",
  "suggestion": "This button can only be found by its text (\"Continue\"), so tests break when
                 the wording changes and do not work in another language. Give it a `data-testid`."
}

The advice names the thing to add in the vocabulary of that platform — data-testid on web, an accessibility identifier on mobile — because that is what the developer reading it will search for.

Capped at ten examples. The point is to show the shape of the problem, not to hand the element list back.

Getting one​

Any crawl produces it. No flag to set.

# Web
POST /api/crawl { "target": "https://your-app.example.com", "engine": "PLAYWRIGHT", "maxDepth": 3 }

# Mobile
POST /api/mobile-apps/{id}/inspect { "crawlMode": true, "maxDepth": 3 }

Where to read it​

In the UI: open the crawl's report. The advisory sits above the element inventory — score, band, and the specific controls to fix, each with the attribute to add.

Via MCP — advise_ui:

advise_ui(job_id="<the UUID from crawl_ui>")

Returns the score, band, counts, and examples with a concrete suggestion per control, for a few hundred tokens rather than the thousands an element inventory costs. This is the web counterpart to advise_swagger, and the same advice applies: call it before you author anything. A page whose controls have no stable identifier will produce brittle tests however carefully they are written, and that is worth knowing before you write fifty of them.

get_crawl_results(job_id) carries the advisory too, so you see the grade without having to ask.

Crawls that finished before 2026-08-17 predate the advisory; advise_ui says so rather than returning an empty verdict. Re-run the crawl to get one.

Did the crawl see enough to judge? — saturation​

A score on its own can mislead, and it misleads in the flattering direction. This is the part worth reading twice.

The score is well-named controls ÷ controls found. Controls the crawl never reached are in neither number. So a crawl that stopped early does not score worse for having missed things — it simply does not count them, and a thin crawl can come back looking better than a thorough one of the same application.

Every crawl therefore carries a second verdict saying whether it ran out of application or ran out of budget:

"saturation": {
  "verdict": "TRUNCATED",
  "mayBackCoverage": false,
  "pagesRead": 20,
  "distinctFound": 4,
  "dryStreakAtEnd": 11,
  "discoveryCurve": [ {"afterPages": 5, "newFound": 4}, … ],
  "stopReason": "BUDGET_EXHAUSTED",
  "summary": "The crawl stopped because it hit its page limit, not because it ran out of app…"
}

The verdicts​

verdictWhat happenedmayBackCoverage
SATURATEDThe last several pages found nothing new. The surface looks exhausted.true
NOT_SATURATEDStill finding new screens when the crawl ended. There is more out there.false
TRUNCATEDIt stopped at its page or depth limit, not because it ran out of app.false
INSUFFICIENT_DATAToo few pages to say either way.false

mayBackCoverage is the field to branch on. When it is false, the score describes the pages that were crawled and nothing more — do not put it in front of a customer as a verdict on their application, and do not feed it into a coverage number.

TRUNCATED is deliberately unforgiving. However flat the discovery curve looked, a crawl that hit its limit is one that was prevented from finding out whether there was more. A flat tail proves nothing when something else stopped you.

Why a deeper crawl often scores LOWER​

This surprises people, so it is worth stating plainly: going deeper usually makes the number go down, and that is the number getting more honest, not the application getting worse.

The first pages of most applications are the polished ones — a login, a dashboard. The admin screen nobody has touched in two years is four clicks in, and it is where the unnamed buttons live. A three-page crawl never meets it.

So treat a score from a truncated crawl as a floor on the good news, not a measurement.

stopReason, and the trap inside it​

stopReasonMeaning
LINKS_EXHAUSTEDRan out of links to follow — the genuine end
BUDGET_EXHAUSTEDHit maxUrls — raise it and run again
DEPTH_LIMITHit maxDepth — pages it had already found were left unopened

DEPTH_LIMIT is called out separately because it is the one that hides. At maximum depth the crawler stops collecting links, so its queue empties — which looks exactly like running out of application. Reported as LINKS_EXHAUSTED it would let a depth-truncated crawl be called SATURATED with a whole tier of the app never opened. It is reported distinctly so that cannot happen.

A worked example​

A real crawl of a demo application, deliberately shallow — maxDepth: 1, maxUrls: 3:

score 22   band HARD_TO_AUTOMATE   9 controls assessed, 2 well-named, 7 need attention

examples[0]:
  type       button
  text       "Login"
  quality    WEAK
  selector   button.oxd-button
  suggestion "This button can only be found by its text ("Login"), so tests break
              when the wording changes and do not work in another language.
              Give it a `data-testid`."

saturation  INSUFFICIENT_DATA     mayBackCoverage: false

Read in order:

  1. band: HARD_TO_AUTOMATE — fewer than half the controls have an identifier a developer chose. Tests written against this page today will break on almost any change.
  2. 2 of 9 — the denominator is interactive controls only. Nine is small because this is a login page, not because the crawler missed things on it.
  3. The example is the deliverable. Not "your selectors are weak" but: this button, found only by the word Login, needs a data-testid. That is a sentence a developer can act on without knowing anything about test automation. It also names the two concrete consequences — a copy edit breaks it, and it never worked in the German build.
  4. mayBackCoverage: false — and this is the honest part. Three pages is not enough to judge an application. The 22 is real for what was seen and must not be reported as the score. Raise maxDepth and maxUrls and run it again.

The right conclusion from this output is "this login page needs work, and we have not yet measured the application" — not "this application scores 22".

What to do with a bad score​

  1. Check mayBackCoverage first. If it is false, crawl deeper before doing anything else. You are about to send developers to fix a sample.
  2. Work the examples list, not the score. Each entry is one attribute on one element. Most teams find the same missing convention repeated — a component library that never forwards data-testid, say — so a handful of fixes moves a lot of controls at once.
  3. Re-crawl and compare. The score is only interesting as a trend. A single number is a conversation; a number that moved is a result.
  4. Do not chase 100. READY at 80 means tests written today survive UI changes. The last twenty points are usually decorative controls nobody will ever write a test against.

Known limits​

It scores what was crawled — see saturation above, and always check mayBackCoverage before quoting a number.

A good score is not a guarantee. It says your controls are addressable, which means tests written against them will not break for naming reasons. It says nothing about whether the tests exist, or whether they assert anything useful.

Naming is one dimension of testability. Timing, animation, iframes, shadow DOM and dynamic ids all make an app harder to automate too. The score is about the one thing that is both measurable from a crawl and squarely in a developer's control.