API Testing with YAML

API test scenarios can be written in YAML for a more readable, portable format. The YAML editor in the UI provides autocomplete for all keywords.

Example Scenario

name: User CRUD Test
baseUrl: "{{baseUrl}}"
auth:
  type: BEARER
  loginEndpoint: /auth/login
  tokenPath: $.accessToken
  username: "{{username}}"
  password: "{{password}}"
steps:
  - name: List users
    method: GET
    path: /api/users
    expect: [200]
    assert:
      - path: $.total
        greaterThan: "0"
  - name: Create user
    method: POST
    path: /api/users
    body:
      username: testuser
      email: test@example.com
    expect: [201]
    extract:
      userId: $.id
  - name: Get user
    method: GET
    path: /api/users/{{userId}}
    expect: [200]
    assert:
      - path: $.username
        equals: testuser
  - name: Delete user
    method: DELETE
    path: /api/users/{{userId}}
    expect: [200]

YAML vs JSON

The same scenario can be defined in JSON (via API) or YAML (via editor). The backend converts between formats automatically.

Feature	YAML	JSON
Readability	Clean, indented	Verbose, nested
Editing	YAML editor with autocomplete	Form-based UI
API	POST /api/scenarios/yaml	POST /api/scenarios
Export	GET /api/scenarios/{id}/yaml	GET /api/scenarios/{id}
Validation	POST /api/scenarios/yaml/validate	Built-in on save

YAML Keywords

The editor provides autocomplete for these keywords:

Root Level

name, description, baseUrl, appTag, auth, steps

Auth Configuration

type, loginEndpoint, tokenPath, username, password, tokenUrl, clientId, clientSecret, scope, keyName, keyValue, location

Auth types: BEARER, BASIC, API_KEY, OAUTH2_CLIENT, NONE

Step Fields

name, method, path, body, headers, expect, assert, extract

Methods: GET, POST, PUT, PATCH, DELETE

Assertions

assert:
  - path: $.fieldName
    equals: "expected value"    # exact match
  - path: $.count
    greaterThan: "0"            # numeric comparison
  - path: $.items
    exists: true                # field exists (use `exists: false` for NOT_EXISTS)
  - path: $.message
    contains: "success"         # substring match
  # any other operator (LESS_THAN, NOT_EQUALS, …) uses the generic form:
  - path: $.responseTime
    operator: LESS_THAN
    expectedValue: "500"

The shorthands equals / contains / exists / greaterThan are sugar; every operator is also expressible with operator: + expectedValue:. Both forms round-trip losslessly through get_scenario_yaml export — extractions and the full assertion {operator, expectedValue} are preserved (release-repo #270).

Unknown keys are flagged

POST /api/scenarios/yaml/validate now emits a warning for any unrecognized step or assertion key (e.g. a typo, or soft/continueOnFailure, which the DSL does not implement) so a silently-ignored key surfaces at authoring time instead of at runtime (release-repo #269).

Variable Extraction

extract:
  userId: $.id                  # JSONPath extraction
  token: $.data.accessToken     # nested path

Use extracted variables in later steps with {{variableName}}.

YAML import is JSONPath-only

The extract: block parses every value as a JSONPath expression. The wizard form additionally supports REGEX and HEADER extraction types — if you need those, build the step in the wizard rather than the YAML editor. Exporting a scenario that contains REGEX/HEADER extractions back to YAML is not currently round-trip safe.

Variable Resolution

The platform substitutes {{variableName}} placeholders at execution time from four sources. Understanding the lookup order helps debug missing-value errors and avoid false-pass tests.

Sources (resolved in this order)

Priority	Source	Variables it provides
1	Caller initialVariables	Anything passed on the execute_scenario request
2	Env target lookup	baseUrl (from the env's active target matching the scenario's appTag)
3	Env credential vault	username, password, token (from the credential by credentialTag — falls back to the env's default credential)
4	Prior step extract: outputs	Any variable name written by an earlier step's extraction block
—	hooks.before outputs	Variables produced by hook scripts (e.g. randomString(8) → userSuffix)

The first three are seeded into scope before step 1 runs so a scenario can reference {{baseUrl}}, {{username}}, {{password}}, and {{token}} without any setup. The fourth grows the scope as steps execute.

Save-time validation (release#159)

POST /api/scenarios/yaml and the MCP create_scenario_from_yaml tool now reject any YAML that references a variable with no source. The validator builds the symbol table from:

• Built-in env-bound names: baseUrl, username, password, token, apiKey, target
• hooks.before declarations
• Every extract: block, ordered by step

…then scans every string value in the YAML for {{name}} tokens. If a name isn't in the table, you get a 400 with a message naming the offending variable and listing what was available:

{
  "valid": false,
  "errors": [
    "step 2: `{{userid}}` references an unknown variable. Known sources: [baseUrl, username, password, token, apiKey, target, userId]. Fix one of: (1) extract this var in an earlier step's `extract:`, (2) declare it in `hooks.before`, (3) use a builtin env-bound var, or (4) fix the typo."
  ]
}

This catches typos ({{userid}} vs {{userId}}), forward references (using a var before the step that extracts it), and copy-paste regressions before the scenario is persisted.

Runtime survivor check (release#159)

If a variable's source resolves at runtime to null (e.g. step N's extraction returned no match for a JSONPath), the substitution leaves the literal {{name}} in the rendered request. The executor scans the resolved URL, body, headers, and assertion expectedValue for any surviving {{ placeholder before calling the HTTP client. If anything survives, the step is failed with a clear message:

Unresolved template variable {{token}} reached the request at step 3.
No matching source in scope. Available variables: [baseUrl, username,
password, userId]. Fix: extract this var in an earlier step, declare
it in hooks.before, set it on the env credential vault, or correct
the typo. Refusing to execute — would have sent the literal string
'{{token}}' to the server, which usually false-passes the assertion.

This catches the runtime gap the static validator can't see: a hook output that silently failed, or a step-N extraction that returned no match leaving step-N+1's reference unresolved.

What this prevents

Before these checks, three common failure modes silently passed:

Failure	Old behavior	New behavior
Typo in extracted var name	{{userid}} substituted as "" — request URL became /api/users/ (root list), assertion $.id equals "{{userid}}" could false-pass	Save rejected with 400 at validate time
Unauthored credential ref	{{token}} substituted as "" — Authorization: Bearer sent, server may accept for testing	Step fails before HTTP call with full available-vars listing
Hook didn't run	Variable empty everywhere it was used, no diagnostic	Step fails on first reference with the unresolved name

Using the YAML Editor

On the Scenario Edit Page

1. Open any API scenario at /scenarios/{id}/edit
2. Click the YAML button in the header
3. View and inspect the scenario in YAML format
4. Click Validate to check syntax
5. Click Form View to switch back

On the API Tests List Page

1. Navigate to API Tests
2. Click the YAML Editor button
3. Write a new scenario in YAML
4. Click Validate then Create from YAML

API Endpoints

# Create scenario from YAML
POST /api/scenarios/yaml?projectId=1&environmentId=1
Content-Type: text/plain

name: My Test
baseUrl: "{{baseUrl}}"
steps:
  - name: Health check
    method: GET
    path: /health
    expect: [200]

# Validate YAML without creating
POST /api/scenarios/yaml/validate
Content-Type: text/plain

# Export existing scenario as YAML
GET /api/scenarios/{id}/yaml