ax-check rule

AXC-C013: Advertised JSON output not verified

Help advertises machine-readable output, but this run did not validate it: the json probe was not enabled, could not be built, or produced no output.

ax-check is a checker being prepared for release. This page documents the rule ahead of that release; see all 50 rules.

Severityinfo
KindObserved. Depends on the program and environment at run time.
Modeax-check probe-cli
Applies toCommand-line programs, run as a subprocess
Pattern tagsnon-interactive, false-success
Fix in one lineRe-run with --probe json on a command that is safe to run with its JSON selector, or check the output by hand.

The help text advertises machine-readable output, such as --json or --format json, but this run did not check that the output parses. The finding is a statement about the run, not about your CLI. It is there so that a clean report never hides a flag that nobody tested.

What it checks

When the help text advertises machine-readable output, ax-check reports this rule unless the json probe ran, exited 0 and printed something on stdout that it could validate. The message gives the reason. There are four:

  • the json probe is opt-in and was not enabled (the default);
  • the help text names machine-readable output, such as --porcelain, that ax-check cannot turn into a JSON selector;
  • the json probe exited with a non-zero status, so there was no output to validate;
  • the json probe printed nothing on stdout.

When the json probe did validate the output, this rule is silent, and AXC-C007 reports output that does not parse.

Why it matters

A JSON flag in help text is a claim. An agent trusts it and parses whatever comes back. Before this rule existed, a help text that mentioned --format json was enough to silence AXC-C006, even when no probe ever ran the selector. A report with no findings then looked like a pass for output nobody had seen. Saying “not verified” keeps the report honest about its coverage.

How to fix

If the command is safe to run with its JSON selector, re-run with --probe json, for example ax-check probe-cli --probe json -- invoicer list. If the selector needs a subcommand to produce data, probe that read-only subcommand. If you cannot run it safely, check the output by hand: invoicer list --format json </dev/null 2>/dev/null | jq ..

Example

Before

$ ax-check probe-cli -- invoicer
  info  AXC-C013 [observed]  Help advertises `--format json`, but its output was not verified: the json probe is opt-in and was not enabled (add --probe json).

After

$ ax-check probe-cli --probe json -- invoicer list
No findings.

How ax-check detects it

ax-check builds the selector from the help text as an argument array: a boolean flag such as --json, a value flag whose choices or description mention JSON (--format json, --output json, -o json), or a selector written out in the text, such as invoicer list --output=json. It passes the flag and its value as separate arguments, never through a shell. The rule then looks at the json probe’s result, if there is one.

Known false positives: a CLI whose JSON selector needs a subcommand, probed at the root. The probe may exit non-zero because no subcommand was given. Probe a read-only subcommand instead. Silence the rule with --disable AXC-C013 once you have checked the output another way.

Safety note: probe-cli executes your program under your own OS account. The json probe runs your command with its JSON selector added, which on some CLIs does the command’s ordinary work, so it is opt-in. The temporary working directory and HOME are not a sandbox. See the “Probe safety” section of the ax-check README for what is and is not isolated.

Sources

  • Guideline: Command Line Interface Guidelines, clig.dev. https://clig.dev/ . Says to display output as formatted JSON if --json is passed and to send messaging to stderr.

Records in the AX evidence register that share a pattern tag with this rule. A shared tag means the record is about the same pattern, not that it tests this rule. Read the evidence class before the number.