Tests

dmx runs tests at the gates of the Dev and Validate loops, before work moves on. In a repository with components, it runs the tests of the components a change can affect. This page explains how it chooses them, how each component's tests are run, and what to do when a test fails for reasons that have nothing to do with the change.

The examples use Acme's repository from Monorepos: ledger, payments and billing-ui, where billing-ui depends on payments and payments depends on ledger.

Which tests runCopy Anchor Link

dmx looks at the files the branch has changed, compared with the integration branch, and picks the tests in three steps:

1. The changed components. Each component that owns a changed file runs its tests.
2. The components that depend on them. Each component that depends on a changed one runs its tests too, and so does each component that depends on those, all the way along.
3. Repo-wide changes. If the branch changed a repo-wide file, one no component owns, dmx runs the test_command from .dmx/config.md. If there's no test_command, it runs every component's tests, because a repo-wide change can affect any of them.

At Acme:

A branch that hasn't changed anything yet counts as a repo-wide change, so a gate never passes without running any tests.

How a component's tests are runCopy Anchor Link

Each component has one command that runs its tests. For most code, dmx works the command out from what's in the component's folder:

To use a different command, set test in the component's component.yaml:

test:
  cmd: make test
  cwd: services/payments

cmd runs through sh -c (on Windows, the sh that Git for Windows installs). cwd is the folder to run it in, relative to the root of the repository; without it, the command runs from the root.

When several components have the same command and folder, it runs once for all of them.

Tests stored outside the packageCopy Anchor Link

Some repositories keep their tests in a top-level tests/ folder rather than next to the code:

packages/
├── ledger/
└── payments/
tests/
├── ledger/
└── payments/

For dmx to find those tests, include the test folder in the component's paths:

paths: ["packages/ledger/**", "tests/ledger/**"]

For a Python component whose own folder holds no tests, dmx then runs the test folders and files in its paths with pytest, from the root of the repository: uv run pytest -q tests/ledger. If it can't find any tests for a component, it reports the component as not configured rather than running nothing.

Running tests the way CI doesCopy Anchor Link

A test command that passes in CI can still fail on a developer's machine, because the CI job sets things that the command doesn't. For example, a CI job might set CI=true, or turn off a check that only matters in production.

Look at the CI job that runs each component's tests, and copy its environment variables and flags into the component's command:

test:
  cmd: "CI=true uv run --all-extras pytest -q -n auto tests/ledger"

When the agent writes test commands during /dmx/init, it reads your CI workflow and does this for you, and tells you which settings it took.

Tests that fail for other reasonsCopy Anchor Link

Sometimes a test fails whatever the change. It fails on the integration branch too, or only on some operating systems, or only when the whole suite runs at once. Left alone, such a test fails the gate for every piece of work that runs it, even work that has nothing to do with it.

Don't turn it off by excluding it from the command; then nobody sees it again. List it as a known failure instead, in the component's test entry, with a comment saying why:

test:
  cmd: go test ./...
  cwd: services/payments
  known_failures:
    - TestSettlementClock          # fails on macOS runners; see PAY-88

The test still runs. When the only tests that fail are known failures, the gate passes, and the result says which known failures failed. Any other failure still fails the gate.

How entries matchCopy Anchor Link

An entry matches any failing test whose name, as the test runner prints it, contains the entry. One entry can therefore cover:

• a single test: tests/test_display.py::test_banner;
• every test in a class: TestEmbeddings;
• every test in a file: tests/durable_exec/test_prefect.py.

dmx recognises failing tests in the output of pytest (including with the pytest-pretty plugin), Jest, Vitest and Go. If a run fails and dmx can't find any failing test names in its output, it says the known failures couldn't be checked, and the gate fails.

Keeping the list honestCopy Anchor Link

Only list failures that the change didn't cause. Two things keep the list from going stale:

• When a listed test passes again, the result says so, so you can remove it.
• dmx check shows every known failure, so they stay visible in review.

For test_command, list known failures in .dmx/config.md:

test_known_failures: [tests/test_lock.py::test_resolution]

Reading the resultsCopy Anchor Link

The gate reports one line for each command it ran, and shows a failing run's output under the component's name:

Tests failed — 1 of 2 runs failed
✓ ledger — go test ./services/ledger/... (exit 0)
✗ payments — go test ./... in services/payments (exit 1)

Some results mean the setup needs attention, not the code. When the Validate loop finds failed checks, it can propose tasks to fix them (see rework), but it never proposes code tasks for these:

All the test runs at one gate share one time limit: the timeout_seconds of the run_tests validator in the loop's configuration, 630 seconds unless you change it.