Skip to main content

API surface coverage

Purpose: answer the question a pass-rate cannot — how much of what we could test actually has a test?

A green suite tells you the tests you wrote are passing. It says nothing about the endpoints with no test at all, and that is usually where the risk is.

The idea​

Nothing here is discovered. Both halves already exist:

  • the API digest already lists every endpoint in the spec,
  • the scenarios already record which endpoint each step calls.

Coverage is one set against the other. It is code coverage, but for the product surface instead of lines of code.

Two levels, and the gap between them is the finding​

Meaning
referencedsome step calls the endpoint
assertedsome step calls it and checks the answer

A step that fires a request and validates nothing has exercised an endpoint without testing it — it passes whatever comes back, including a 200 containing nonsense.

On most projects the second number is a good deal lower than the first, and those endpoints are listed separately as calledButNotChecked. That list is usually more actionable than the uncovered one, because the test already exists and only needs an assertion.

What counts as checking the answer: a response assertion, an advanced assertion, an extraction, or an explicitly chosen status code. A bare expectedStatusCodes: [200] does not — that is the default every step is born with, so it is not evidence anyone decided anything.

Getting it​

GET /api/coverage/api?appTag=demo&environmentId=1&projectId=1

Via MCP:

api_coverage(app_tag="demo", environment=1, project=1)
{
  "status": "OK",
  "totalEndpoints": 19,
  "referenced": 12,
  "asserted": 7,
  "referencedPct": 63.2,
  "assertedPct": 36.8,
  "uncovered": [
    {"endpoint": "DELETE /api/users/{id}", "method": "DELETE",
     "path": "/api/users/{id}", "operationId": "deleteUser"}
  ],
  "calledButNotChecked": [
    {"endpoint": "POST /api/users", "calledBy": ["Create user"]}
  ],
  "summary": "12 of 19 documented endpoints are called by a test, but only 7 have a test that
              checks the answer — 5 are called and never verified, which passes whatever the
              endpoint returns. 7 have no test at all."
}

Every gap row carries the method, path and operationId, so the missing scenario can be written from the row without opening the spec.

Matching a test to an endpoint​

The spec says /api/users/{id}. A test says /api/users/1, or /api/users/{{userId}}, or a UUID. All the same endpoint, and a literal comparison would score every one of them as uncovered.

Collapsed to a parameter: an OpenAPI template ({id}), one of our variable references ({{userId}}), a segment of pure digits, a UUID.

Deliberately not collapsed: anything else. v1 stays v1; /api/users never becomes /api/*. Being strict here matters more than being clever — a matcher that collapses too much reports endpoints as covered when nothing tests them, and a coverage number that errs upward is worse than none at all, because it is the direction nobody double-checks.

The unit is path + method. GET /api/users and POST /api/users are two endpoints, as the spec lists them.

What it will not do​

It never counts an endpoint the spec does not list. Coverage is of the documented surface. A test against an undocumented endpoint cannot raise the percentage — letting it would hide the fact that the endpoint is missing from the spec, which is its own problem.

No digest is not 0%. If no spec is registered you get status: NO_DIGEST and no percentage. "There is no spec" and "nothing is tested" are opposite problems, and reporting the first as the second sends people to write tests when what they need is to register a spec.

A step whose path cannot be read is reported, not swallowed. It appears as a warning on the response rather than quietly depressing the number.

Known limits​

A number in the path is assumed to be an id. /api/2024/report normalizes the same way /api/users/2024 does. Ids are overwhelmingly the common case, so digits win — but an API that puts a year or a version number in a path segment will have those endpoints merged. Stated here rather than hidden.

Coverage is only as complete as the spec. An endpoint nobody documented is invisible to this number, in both directions.

It says nothing about quality. An asserted endpoint has an assertion. Whether that assertion is worth anything is a different question, and one this number does not pretend to answer.