Skip to main content

Progressive narrowing — the resolver contract

Every Proofarc MCP tool that takes a name (project, environment, scenario, UI test, pipeline) accepts either a string name or an integer id. When a name is ambiguous, unknown, or missing, the tool does not raise — it returns a structured response the LLM can relay back to the user as a follow-up question.

The conversation narrows on every round until there's a clean execute.

The response shape​

When resolution can't proceed cleanly:

{
  "reason":     "ambiguous" | "not_found" | "missing_required",
  "ask_user":   "Which environment? Available: dev, staging, prod.",
  "candidates": [
    {"id": 11, "name": "dev"},
    {"id": 12, "name": "staging"},
    {"id": 13, "name": "prod"}
  ],
  "scoped_by":  {"project": "payments"}
}

Field-by-field:

FieldMeaning
reasonWhy we're not executing yet. ambiguous = multiple matches; not_found = no match; missing_required = caller didn't pass the field at all.
ask_userA literal one-line question. Claude relays this back to the human verbatim.
candidatesThe narrowed option set. Pre-filtered by every constraint the caller already pinned. Capped at 25 entries.
scoped_byWhat's already pinned. E.g. {"project": "payments"} means candidates are already restricted to that project.

The 1–3 turn convergence​

A request like "run the smoke tests on staging" typically resolves in 1–3 turns:

Turn 1 — bare request​

Claude calls:

run_by_tag(tags=["smoke"])

Server returns:

{
  "reason": "missing_required",
  "ask_user": "Which environment do you want to run these tests in? Available: dev, staging, prod.",
  "candidates": [
    {"id": 11, "name": "dev"},
    {"id": 12, "name": "staging"},
    {"id": 13, "name": "prod"}
  ],
  "scoped_by": {}
}

Claude asks the user: "Which environment?"

Turn 2 — env added, project still ambiguous​

User: "staging". Claude calls:

run_by_tag(tags=["smoke"], environment="staging")

If smoke-tagged tests live across multiple projects, server returns:

{
  "reason": "ambiguous",
  "ask_user": "Tag(s) ['smoke'] match tests across multiple projects: payments, auth. Which project do you want to run in?",
  "candidates": [
    {"id": 1, "name": "payments"},
    {"id": 2, "name": "auth"}
  ],
  "scoped_by": {}
}

Turn 3 — pinned, executes​

User: "payments". Claude calls:

run_by_tag(tags=["smoke"], environment="staging", project="payments")

Server returns:

{
  "dry_run": false,
  "tags": ["smoke"],
  "environment": "staging",
  "project": "payments",
  "queued": [
    {"kind": "api_scenario", "id": 204, "name": "Login flow", "executionId": "...", "status": "QUEUED"},
    {"kind": "api_scenario", "id": 205, "name": "Cart", "executionId": "...", "status": "QUEUED"}
  ],
  "skipped": []
}

Why "pre-scoped" matters​

Candidates always reflect everything the caller has already pinned. If turn 2 already constrained to environment="staging", the project candidates returned on turn 3 are only those with smoke tests in staging. The set shrinks each round; it never widens. The LLM doesn't get the universe dumped on it every time.

This is the progressive narrowing rule: every reply gives the LLM exactly the choices that would advance the search, scoped by what's already been decided.

Resolution paths in detail​

The resolver accepts a value and goes through this decision tree:

value is None
  └─ → return {reason: "missing_required", candidates: <all in scope>}

value is int (or string of digits)
  └─ → coerce to int and return it (trust the caller)

value is str (name)
  ├─ exactly one item has that name (case-insensitive)
  │    └─ → return its id
  ├─ multiple items have that name
  │    └─ → return {reason: "ambiguous", candidates: <those matches>}
  ├─ no exact match, but exactly one item contains the substring
  │    └─ → return its id ("pay" → "payments")
  ├─ no exact match, multiple substring matches
  │    └─ → return {reason: "ambiguous", candidates: <substring matches>}
  └─ no matches at all
       └─ → return {reason: "not_found", candidates: <all in scope>}

When a project is pinned (via the project argument or by an earlier resolve), every subsequent lookup runs against the project-filtered snapshot, not the full instance. That's how Login flow becomes unambiguous as soon as the user says "in payments."

Tools that follow the contract​

Every name-taking tool:

  • list_project_applications(project=…)
  • list_environments(project=…)
  • find_scenarios(project=…)
  • find_ui_tests(project=…)
  • list_pipelines(project=…)
  • create_api_scenario(project=…, environment=…)
  • execute_scenario(scenario=…, environment=…, project=…)
  • run_ui_test(test=…, environment=…, project=…)
  • run_by_tag(tags, environment=…, project=…)
  • run_pipeline(pipeline=…, environment=…, project=…)

Tools that don't take names (because execution ids are server-minted opaque tokens) are unchanged:

  • get_execution_status(execution_id, kind)
  • summarize_execution(execution_id, kind)

Caching​

The resolver is session-scoped. Within one MCP conversation, the /api/projects, /api/environments, /api/scenarios, /api/ui-tests, and /api/pipelines snapshots are each fetched once and reused for all subsequent lookups. A 5-turn conversation that resolves the same project multiple times produces a single GET /api/projects against the backend.

See also​