Skip to main content

Generation vs Composition — the boundary

Purpose: keep a hard line between what the platform generates (deterministic scaffolding) and what an AI agent or human composes (reasoning). This prevents a recurring mistake: re-implementing agent reasoning inside a rule-based generator.

This is the durable rationale behind create_scenario_from_swagger emitting literal {id} being working-as-designed (release#420).

The two layers​

LayerWhoExamples
Generation — deterministic, rule-basedPlatform codeparse the OpenAPI spec; enumerate endpoints; build the API/crawl digest; emit step skeletons (method, path, expected codes); crawl a page and extract selectors; validate YAML.
Composition — requires reasoningAI agent or humaninfer cross-operation data-flow (which call produces an id another call consumes); order operations (create before get/update/delete); synthesize request bodies under constraints; add negatives & assertions (expect 404 after delete, expect 403 for the wrong actor).

Rule: Generation must NOT try to do Composition. A rule-based generator cannot infer that POST /api/users produces the id that GET /api/users/{id} consumes — that is reasoning, and reasoning belongs to the agent or the human. Pushing it into the generator means re-inventing the agent, badly.

Why this is real, not theoretical​

create_scenario_from_swagger against user-service /v3/api-docs emits:

GET /api/users/{id}    <- literal {id}, sent as %7Bid%7D -> HTTP 400
PUT /api/users/{id}
DELETE /api/users/{id}
GET /api/users
POST /api/users        <- no request body

Checked against the live spec: 0 of 3 path params have an example, 0 of 5 request bodies have an example (only User.username/email/role carry property examples). So "fill {id} from the spec example" is impossible here — there is nothing to fill from. The only way to a runnable {id} is to chain: create a user, extract $.id, reuse it. That requires reasoning → agent/human.

So create_scenario_from_swagger is a rule-based bulk scaffold / fallback for non-LLM clients, by design. It is not broken; it is not meant to produce runnable chained tests on its own.

The intended path (Composition)​

parse_swagger                      # generation: structure
  -> agent composes chained YAML   # composition: data-flow, ordering, bodies, negatives
  -> validate_scenario_yaml        # generation: check
  -> create_scenario_from_yaml     # generation: persist
  -> execute_scenario              # run -> green

The agent (or a human) turns the scaffold into a runnable contract. Don't start from a blank page: the CONTRACT_ANCHOR playbook ships a guided template you fill in — find it with find_playbooks(intent="contract test for an API"), read it with get_playbook(id), and instantiate_playbook(id) returns the chained skeleton as its artifactTemplate.

The golden example​

The reference output of that composition (proven green 5/5 against user-service):

# Golden CONTRACT_ANCHOR example — the COMPOSED, runnable output (not raw swagger-gen).
# Shows the reasoning a rule-based generator cannot do: data-flow (extract $.id -> reuse),
# operation ordering (create first), request bodies, and a negative (404 after delete).
name: user-crud-contract
baseUrl: "http://user-service:8089"
appTag: user-service
credentialTag: admin
steps:
  - name: create user
    method: POST
    path: /api/users
    body: '{"username":"anchor1","email":"anchor1@example.com","password":"anchorpass1","role":"USER"}'
    expect: [200, 201]
    extract:
      userId: $.id            # <-- composition: capture the id this call PRODUCES
  - name: get user
    method: GET
    path: /api/users/{{userId}}   # <-- composition: reuse it (not literal {id})
    expect: [200]
  - name: update user
    method: PUT
    path: /api/users/{{userId}}
    body: '{"username":"anchor1","email":"anchor1b@example.com","password":"anchorpass2","role":"USER"}'
    expect: [200]
  - name: delete user
    method: DELETE
    path: /api/users/{{userId}}
    expect: [200, 204]
  - name: get deleted (404)        # <-- composition: negative assertion proving the delete
    method: GET
    path: /api/users/{{userId}}
    expect: [404]

The same file ships as a downloadable sample: /samples/contract-anchor-user-crud.yaml.

Setting up the sample target​

The contract above runs against user-service. Set it up once (no UI needed) following Demo 1 — Add Targets, or run the PROJECT_SETUP playbook (find_playbooks(intent="set up a project")): create the project + REST app + environment + admin credential + user-service target, then execute_scenario the contract.

Guardrails for contributors​

  • ✅ Improve generation: better endpoint enumeration, spec parsing, digest quality, selector extraction, and flag what can't be auto-filled (missing examples).
  • ✅ Improve the agent's raw material: surface examples, schemas, and constraints so composition is easier.
  • ❌ Do not add cross-call data-flow inference, operation re-ordering by dependency, or constraint-aware body synthesis to the rule-based generator. That is composition — leave it to the agent.
  • If a generated step can't be made runnable deterministically, skip/flag it — don't emit a literal placeholder that 400s and fail-fasts the scenario.
  • create_scenario_from_swagger — the rule-based scaffold/fallback (this doc is its rationale)
  • Creating Scenarios · YAML Format
  • CONTRACT_ANCHOR playbook + the golden example above (the composed reference)