Authoring tests as an AI agent

If you're an LLM (Claude, GPT, etc.) authoring tests against this platform via MCP: use the building-block path. Crawl the site or parse the spec, read the inventory, compose the YAML yourself, validate it, and persist it. Do not use rule-based composer tools like generate_ui_test_from_crawl (deprecated — see release#156) or rely on create_scenario_from_swagger's heuristic bulk-generation for production-quality tests. Those exist for non-LLM clients and produce uniformly mediocre output that an LLM can outperform by reasoning over the same source material.

The principle (release#156)

The platform's responsibility for AI-driven test authoring stops at exposing building blocks. Composition is the agent's job. The platform must never compose final test YAML on behalf of an LLM client.

Layer	Owns
Platform / API	Crawl pages → element inventory. Parse Swagger → endpoint catalog. Validate YAML against schema. Persist tests. Execute tests. Resolve credentials / environments / targets.
Agent (LLM)	Read the inventory. Map user intent to a step sequence. Choose selectors with awareness of the actual DOM. Compose assertions that catch real bugs. Insert smart waits and error recovery.
User	Provide natural-language intent. Provide context the agent can't infer (credentials, success criteria, test-data preferences).

Canonical flow — UI tests

ensure_crawl_for_authoring(url="https://app.example.com/login")
  ⇒ {jobId, fresh, pages: [{url, title, elements: [...]}, ...]}

# YOU compose the YAML from those elements:
yaml = """
steps:
  - action: NAVIGATE_TO
    value: "{{baseUrl}}/login"
  - action: WAIT_FOR_VISIBLE
    selector: "#username"
  - action: SEND_KEYS
    selector: "#username"
    value: "{{username}}"
  - action: SEND_KEYS
    selector: "#password"
    value: "{{password}}"
  - action: CLICK
    selector: "button[type=submit]"
  - action: VALIDATE_URL
    expectedText: "/dashboard"
  - action: VALIDATE_VISIBLE
    selector: ".user-menu"
"""

validate_ui_test_yaml(yaml_text=yaml)
  ⇒ {valid: true, errors: []}    # iterate until valid if needed

create_ui_test_from_yaml(
  name="Login flow",
  yaml_text=yaml,
  project="User Management Platform",
  tags="smoke,login")
  ⇒ {id: 123, projectId: 2, status: "ACTIVE"}

run_ui_test(test="Login flow", environment="development",
            wait=true, summarize=true)
  ⇒ {jobId, status: "COMPLETED", summary: "All 7 steps passed in 4.1s"}

Canonical flow — API scenarios

parse_swagger(url="https://api.example.com/v3/api-docs")
  ⇒ {info, endpoints: [{method, path, operationId, ...}, ...], baseUrl}

# YOU compose the scenario YAML — pick the endpoints that exercise the
# real user flow, set up variable chaining, write assertions that catch
# bugs (not just status-code checks):
yaml = """
name: User CRUD round-trip
baseUrl: "{{baseUrl}}"
steps:
  - name: Create user
    httpMethod: POST
    endpointPath: /api/users
    requestBody:
      email: "demo+{{$randomString(8)}}@example.com"
    expectedStatusCodes: [201]
    extractions:
      - variableName: userId
        extractionType: JSONPATH
        extractionExpression: $.id
  - name: Read user
    httpMethod: GET
    endpointPath: /api/users/{{userId}}
    expectedStatusCodes: [200]
    responseAssertions:
      - type: JSONPATH
        expression: $.email
        operator: EQUALS
        expectedValue: "demo+{{$randomString(8)}}@example.com"
  - name: Delete user
    httpMethod: DELETE
    endpointPath: /api/users/{{userId}}
    expectedStatusCodes: [204]
"""

validate_scenario_yaml(yaml_text=yaml)
  ⇒ {valid: true, errors: []}

create_scenario_from_yaml(
  yaml_text=yaml,
  project="User Management Platform",
  environment="development",
  tags="smoke,crud")
  ⇒ {id: 73}

execute_scenario(scenario="User CRUD round-trip",
                 environment="development",
                 wait=true, summarize=true)
  ⇒ {executionId, status: "COMPLETED", summary: "3/3 steps passed"}

What each building block returns

`ensure_crawl_for_authoring(url, ...)`

The element inventory across all crawled pages. Cached for 24h, so repeat calls for the same URL are free. Use force_refresh=True to skip the cache.

Per-element shape:

{
  "semanticName": "Username field",
  "primarySelector": "#username",
  "tag": "input",
  "type": "text",
  "id": "username",
  "name": "username",
  "placeholder": "Enter username",
  "text": "",
  "xpath": "//input[@id='username']",
  "chrome": false
}

primarySelector is the platform's preferred selector — usually the CSS one, sometimes XPath when no stable CSS exists. chrome: true means the element is part of the site shell (nav, footer) and shows up on many pages; filter these out for most action-flow tests.

`parse_swagger(url=... or content=...)`

The endpoint catalog from a Swagger / OpenAPI spec, plus the spec's declared baseUrl. Use this catalog to decide which endpoints to include in a scenario, what request bodies look like, and what response shapes to assert against.

`validate_ui_test_yaml(yaml_text)` / `validate_scenario_yaml(yaml_text)`

Schema check against the platform's parser. Returns {valid, errors}. Cheap loop: draft → validate → fix → validate → persist.

`create_ui_test_from_yaml(name, yaml_text, ...)` / `create_scenario_from_yaml(...)`

Persists the YAML you composed. Validates server-side too, so a non-validated YAML will be rejected here.

`run_ui_test(test, ...)` / `execute_scenario(scenario, ...)`

Executes. With wait=true, summarize=true you get a one-call human-readable summary back, no extra polling.

What violates the principle (and what to do instead)

Tool that composes for you	Use these blocks instead
`~~generate_ui_test_from_crawl~~` (deleted 2026-05-24, release#156)	`ensure_crawl_for_authoring` → compose YAML → `validate_ui_test_yaml` → `create_ui_test_from_yaml`
`create_scenario_from_swagger` (still exists for non-LLM clients)	`parse_swagger` → compose YAML → `validate_scenario_yaml` → `create_scenario_from_yaml`

The composer tools aren't bad code — they just aren't the right layer for an LLM client. Their rule-based selector matching and intent parsing can't reason about a page the way you can. An LLM picking selectors from the same crawl will produce better tests, every time.

Worked example — Portnov e-commerce smoke test

2026-05-24 session, verified end-to-end:

Crawl: 737f2a15-9dd0-495e-939f-c743aee160b4 — 10 pages, 397 locators on store-qa.portnov.com.
Agent composed a 16-step iPhone add-to-cart smoke test using the inventory + user intent.
Validated via validate_ui_test_yaml — passed on second iteration after fixing WAIT_FOR_URL → VALIDATE_URL (LLM caught + fixed its own typo from the validator's error response).
Persisted as UI test id=123 via create_ui_test_from_yaml.
Executed on Playwright via run_ui_test: 11/16 steps passed on first run against the live e-commerce site.

This is the pattern the platform treats as canonical. The 5 of 16 failed steps were genuine UI issues for the agent (or its human partner) to triage — not platform problems.

The principle (release#156)​

Canonical flow — UI tests​

Canonical flow — API scenarios​

What each building block returns​

ensure_crawl_for_authoring(url, ...)​

parse_swagger(url=... or content=...)​

validate_ui_test_yaml(yaml_text) / validate_scenario_yaml(yaml_text)​

create_ui_test_from_yaml(name, yaml_text, ...) / create_scenario_from_yaml(...)​

run_ui_test(test, ...) / execute_scenario(scenario, ...)​

What violates the principle (and what to do instead)​

Worked example — Portnov e-commerce smoke test​