Data-Driven Testing

Run the same API scenario once per row of input data. A data-driven set is a list of rows bound to a scenario; each row's columns are in scope as {{column}} in the scenario's steps, and you get per-row pass/fail.

Two separate stores — don't confuse them

Data-driven sets (/api/datasets, this page) run an API scenario per row — rows are stored in the database. This is distinct from upload_test_data (/api/test-data), which stores a CSV in object storage to feed Artillery performance tests. The two are intentionally separate.

MCP workflow (recommended)

# 1. Create a set — rows are flat {column: value} objects, bound to a scenario
create_data_driven_set(
    scenario="User CRUD",              # name or id — REQUIRED (a set runs a scenario)
    name="registration-data", version="v1.0",
    description="valid + invalid registration rows",
    tags="registration,smoke",         # optional, comma-separated, for grouping/search
    rows=[
        {"username": "john_doe",  "email": "john@example.com", "role": "USER"},
        {"username": "jane_admin","email": "jane@example.com", "role": "ADMIN"},
        {"username": "",          "email": "invalid",          "role": "UNKNOWN"},
    ],
)

# 2. Run it — the bound scenario executes once per row
run_data_driven(name="registration-data", version="v1.0", environment="dev")

# 3. Read per-row results
get_data_driven_execution(execution=<id from step 2>)

A set's identity is (name, version) — you always refer to it by both, never a numeric id. find_data_driven_set(name, version), list_data_driven_sets(tag=…) and delete_data_driven_set(name, version, confirm=true) round out the surface.

Rules

Rule	Detail
name + version + description	all required
version format	must be v<major>.<minor> (e.g. v1.0, v0.0) — else 400
(name, version) unique	a duplicate is rejected with 409 (bump the version instead)
tags	optional, comma-separated; each tag is letters/digits/-/_, ≤30 chars, ≤10 tags
project	optional — data-driven sets are shareable across projects

Declaring columns in the scenario (dataColumns)

A data-driven scenario references its dataset's columns as {{column}} — but the author-time validator rejects unknown variables, so declare the columns at the top of the scenario YAML:

name: create-users-from-rows
baseUrl: "{{baseUrl}}"              # or appTag: — every scenario needs one host source
dataColumns: [email, username]      # declares the dataset columns as known variables
steps:
  - name: create
    method: POST
    path: /api/users
    body:
      email: "{{email}}"            # substituted per row at run time
      username: "{{username}}"
    expect: [201]                   # required on every step

Without dataColumns:, a {{column}} reference errors at create time as an unknown variable (that's deliberate — it still catches typos: a declared email doesn't make {{emial}} valid).

Two fields above are easy to leave out of a data-driven scenario because neither has anything to do with the rows, and the validator rejects the scenario without them:

• baseUrl or appTag — where the requests go. appTag resolves the host from the environment's target at run time; baseUrl can be concrete or {{templated}}.
• expect on every step — the status codes that count as success. Without it any HTTP response passes, so a run of 200 rows can come back green against a server returning 500s.

Creating a Data Set (REST)

POST /api/datasets
{
  "name": "User Registration Data",
  "version": "v1.0",
  "description": "Test data for user creation scenarios",
  "scenarioId": 1,
  "tags": "registration,smoke",
  "entries": [
    {"testName": "valid user",  "variables": {"username": "john_doe",  "email": "john@example.com", "role": "USER"}},
    {"testName": "admin user",  "variables": {"username": "jane_admin","email": "jane@example.com", "role": "ADMIN"}},
    {"testName": "invalid",     "variables": {"username": "",          "email": "invalid",          "role": "UNKNOWN"}}
  ]
}

Binding to a scenario

A set is bound to its scenario at create time (scenarioId / the scenario arg) — that's the scenario every row runs. Run behavior is controlled on the set:

Field	Description
scenarioId	the scenario each row executes (required)
executionMode	SEQUENTIAL (default) or PARALLEL
stopOnFailure	stop after the first failing row (true) or run them all (false, default)

At run time, run_data_driven / POST /api/datasets/{id}/execute accepts the usual per-run overrides — environmentId, credentialTag, baseUrlOverride, targetId, tags.

Using Data Set Variables

Reference data set fields in your steps with {{fieldName}}:

{
  "stepOrder": 1,
  "name": "Create user from data set",
  "httpMethod": "POST",
  "endpointPath": "/api/users",
  "requestBody": "{\"username\":\"{{username}}\",\"email\":\"{{email}}\",\"role\":\"{{role}}\"}",
  "expectedStatusCodes": [201]
}

Execution Flow

When you execute a data-driven scenario:

1. The scenario runs once per data set entry
2. Each run substitutes the entry's values into {{variables}}
3. Results tracked per entry — see which data inputs pass or fail
4. The overall scenario status reflects all entries

Results

Each data set entry produces its own execution result:

Entry 1: john_doe / john@example.com / USER → PASSED (7/7 steps)
Entry 2: jane_admin / jane@example.com / ADMIN → PASSED (7/7 steps)
Entry 3: test_viewer / viewer@example.com / VIEWER → PASSED (7/7 steps)
Entry 4: (empty) / invalid / UNKNOWN → FAILED (step 1: validation error)

Use Cases

• Boundary testing — empty strings, long strings, special characters
• Role-based testing — same flow with different user roles
• Multi-tenant testing — same API, different tenant credentials
• Negative testing — invalid inputs that should fail validation