Storybook A11y

Accessibility testing workflow with Chromatic, Storybook/Vitest, and the Datadog inventory.

Automated accessibility (a11y) testing for the design system, starting with @prepared911/ui. Everything runs on the same axe-core engine via @storybook/addon-a11y — there is no second test stack to maintain.

Quick Reference

The pipeline has three layers, each with a distinct role:

Layer	Tool	Purpose
Story-level testing	`@storybook/addon-a11y` + Vitest	Run axe per story in a real browser; surface violations in dev, gate CI on opted-in stories
Observability	`a11y:inventory:ui-core` + Datadog	Batch-audit all ui-core stories; ship violation counts as Datadog metrics for trend tracking
Visual regression	Chromatic	Snapshot stories, block PRs on unreviewed visual changes, and report a11y for opted-in stories

Which do I care about?

Writing a component → check the A11y panel in Storybook locally.
Opening a PR → Chromatic is the visual gate for stories opted in with WithSnapshot (unreviewed visual diffs fail; axe debt does not).
Tracking debt over time → the Datadog dashboard below.

Environment Flags

Three flags control a11y behavior. All are inlined into the Vite bundle at build time (via define in .storybook/main.ts and vitest.config.ts), so they must be set before Vitest/Storybook starts. They resolve through resolveA11yTestMode() in @prepared911/util-storybook-helpers.

Flag	Purpose	Used in
`STORYBOOK_A11Y_TODO=1`	Downgrades every mode to `"todo"` — violations show as warnings but never fail tests	`test:storybook:ci` (the standard CI job), so existing debt never blocks PRs
`STORYBOOK_A11Y_AUDIT=1`	Forces the global default from `"off"` to `"error"` — every loaded story runs axe and fails on any violation	`a11y:inventory:ui-core` audit script
`STORYBOOK_TEST_SCOPE=ui-core`	Narrows the loaded story set to `packages/ui-core/src/` only, keeping audit runs fast	`a11y:inventory:ui-core` audit script

Mode precedence (highest → lowest):

STORYBOOK_A11Y_TODO=1 — forces everything to "todo", so nothing errors or blocks.
Story-level withA11yTestingError — enables "error" for the wrapped story/file.
STORYBOOK_A11Y_AUDIT=1 — forces the global default to "error" (used with STORYBOOK_TEST_SCOPE to constrain scope).
Global default "off" (set in preview.ts).

Workflows

Local development

A11y tests are off by default globally. Opt in per story or per file with the helpers below, then run Storybook normally and check the A11y panel in the sidebar:

CI — Storybook Vitest

Runs in .github/workflows/ci.yaml (storybook-test aggregate over four shards). Executes all stories in browser mode (Chromium via Playwright) with STORYBOOK_A11Y_TODO=1, so any story opted in with withA11yTesting* — even in "error" mode — never blocks a merge:

Full audit — ui-core inventory

Runs axe across all ui-core stories in "error" mode and produces a human-readable report plus a Datadog metrics CSV:

Internally this:

Runs Vitest with STORYBOOK_A11Y_AUDIT=1 STORYBOOK_TEST_SCOPE=ui-core --reporter=json --outputFile=a11y-inventory/vitest-results.json.
Writes raw results to a11y-inventory/vitest-results.json.

Vitest 5's json reporter writes to a file (.vitest/json/output.json by default) instead of stdout, so piping its output no longer works — always pass --outputFile, or set reporters: [["json", { stdout: true }]] to print it. 3. Calls scripts/format-a11y-inventory.mjs → produces report.md + summary.json. Open a11y-inventory/report.md to review findings grouped by component, rule, and impact. To format the results for Datadog locally:

CI — merge_group inventory

Runs automatically via .github/workflows/storybook-a11y-inventory.yaml:

Trigger: merge_group (only when turbo/packages/ui-core/src/** changed), or manual workflow_dispatch.
Non-blocking: continue-on-error: true and excluded from merge-gatekeeper's required checks — a failure here never blocks a merge.
What it does: a11y:inventory:ui-core → a11y:format-datadog-metrics → uploads datadog-metrics.csv to Datadog via int128/send-datadog-action.

Chromatic

Runs on relevant non-draft PRs via .github/workflows/chromatic-storybook-ci.yaml. Opt stories in with WithSnapshot. Local publishes: pnpm chromatic.

Visual regression: captures opted-in story snapshots and blocks the PR until a reviewer accepts visual diffs. TurboSnap (onlyChanged: true) only re-snapshots affected stories; a full run is forced when design-token SCSS (packages/ui-core/src/styles/) or apps/storybook/.storybook/globals.scss changes. Snapshots auto-accept on trunk.
Accessibility tests: the build's Accessibility tab reports axe violations for snapshotted stories. The required GitHub status check is the Actions aggregate named Chromatic, not Chromatic's external UI Review status.

Chromatic is complementary to the axe audit: it catches visual regressions (layout, color, contrast) that axe misses, while axe catches semantic/structural issues (missing labels, ARIA roles, keyboard navigation) that visual diffing misses.

Datadog dashboard

Prepared / ui-core a11y tracks violation trends from the merge_group inventory: total violations over time, severity mix, top components, top axe rules, and a per-PR/SHA table. A few things to know when reading it:

Metrics are gauges — each run overwrites, so widgets use max/last per bucket, never sum.
Data is sparse by design: it only lands on merge_group runs that touch turbo/packages/ui-core/src/**.
These ui-core-only counts intentionally differ from Chromatic's totals — Chromatic spans all snapshotted ui-* packages across light and dark modes.

Metrics

Namespace: prepared.ui_core.a11y (all gauges).

Metric	Extra tags	Description
`violations.total`	—	Total axe violation count across ui-core stories
`failed_stories`	—	Stories with at least one violation
`passed_stories`	—	Stories that passed axe cleanly
`unattributed_failures`	—	Stories that failed for non-a11y reasons (axe never ran)
`errored_stories`	—	Stories where the axe runner itself errored
`violations_by_rule`	`rule:<axe-rule-id>`	Count per axe rule (e.g. `color-contrast`, `button-name`)
`violations_by_component`	`component:<folder>`, `component_path:<path>`	Count per ui-core component folder
`violations_by_impact`	`impact:critical\|serious\|moderate\|minor`	Count per axe impact level

Every row is also tagged with environment:github-actions, service:storybook-a11y, package:ui-core, repository:prepared911/prepared911, plus pr:, sha:, and ref: per run.

Note: ref currently ships empty; filter and group by pr/sha instead.

Output files

All audit output is written to turbo/apps/storybook/a11y-inventory/ (gitignored):

File	Produced by	Contents
`vitest-results.json`	`a11y:inventory:ui-core`	Raw Vitest JSON output
`summary.json`	`format-a11y-inventory.mjs`	Parsed counts: total, by rule, by component, by impact
`report.md`	`format-a11y-inventory.mjs`	Human-readable tables — open this to review findings
`datadog-metrics.csv`	`format-datadog-metrics.mjs`	CSV for `int128/send-datadog-action`

Opt-in helpers

Located in turbo/packages/util-storybook-helpers/src/with-a11y-testing.ts, imported from @prepared911/util-storybook-helpers:

Mode summary:

Mode	Storybook UI	Vitest CLI	CI
`"off"`	No a11y results	Passes	No effect
`"todo"`	Violations shown as warnings	Passes	Never blocks
`"error"`	Violations shown as errors	Fails	Blocks unless `STORYBOOK_A11Y_TODO` is set

STORYBOOK_A11Y_TODO=1 is currently set in CI so no story blocks PRs, giving the team time to fix existing violations incrementally. Per component:

Run pnpm a11y:inventory:ui-core and open a11y-inventory/report.md to find the component's violations.
Fix the violations in the component or its stories.
Upgrade the story from withA11yTesting → withA11yTestingError to lock in the clean state and get CI protection going forward.
Repeat until every ui-core component is on withA11yTestingError.

Once all ui-core stories are on "error" mode, remove STORYBOOK_A11Y_TODO from test:storybook:ci to fully gate CI. Watch the Datadog dashboard to confirm the trend is falling.

Troubleshooting

Stale results / cached bundle — clear the Vite dep cache: pnpm storybook:clean (rm -rf node_modules/.vite), then re-run.
Audit reports 0 violations on a first run — treat a suspiciously clean result as a signal to inspect the raw vitest-results.json, not a result to trust. Violations in "error" mode throw before task.meta.reports is populated, so the parser recovers them from Vitest failure messages; a parser regression can silently under-count.
A story fails for a non-a11y reason — it shows up under unattributed_failures, not the violation counts. Check the story's play function.

Testing strategy / Testing Feature Flags

CI/CD & deployment / CI/CD Overview

On this page

CI — Storybook Vitest

Full audit — ui-core inventory

CI — merge_group inventory

Burn-down (a11y debt)

Troubleshooting

Storybook A11y

Accessibility testing workflow with Chromatic, Storybook/Vitest, and the Datadog inventory.

Quick Reference

The pipeline has three layers, each with a distinct role:

Layer	Tool	Purpose
Story-level testing	`@storybook/addon-a11y` + Vitest	Run axe per story in a real browser; surface violations in dev, gate CI on opted-in stories
Observability	`a11y:inventory:ui-core` + Datadog	Batch-audit all ui-core stories; ship violation counts as Datadog metrics for trend tracking
Visual regression	Chromatic	Snapshot stories, block PRs on unreviewed visual changes, and report a11y for opted-in stories

Which do I care about?

Writing a component → check the A11y panel in Storybook locally.
Opening a PR → Chromatic is the visual gate for stories opted in with WithSnapshot (unreviewed visual diffs fail; axe debt does not).
Tracking debt over time → the Datadog dashboard below.

Environment Flags

Flag	Purpose	Used in
`STORYBOOK_A11Y_TODO=1`	Downgrades every mode to `"todo"` — violations show as warnings but never fail tests	`test:storybook:ci` (the standard CI job), so existing debt never blocks PRs
`STORYBOOK_A11Y_AUDIT=1`	Forces the global default from `"off"` to `"error"` — every loaded story runs axe and fails on any violation	`a11y:inventory:ui-core` audit script
`STORYBOOK_TEST_SCOPE=ui-core`	Narrows the loaded story set to `packages/ui-core/src/` only, keeping audit runs fast	`a11y:inventory:ui-core` audit script

Mode precedence (highest → lowest):

STORYBOOK_A11Y_TODO=1 — forces everything to "todo", so nothing errors or blocks.
Story-level withA11yTestingError — enables "error" for the wrapped story/file.
STORYBOOK_A11Y_AUDIT=1 — forces the global default to "error" (used with STORYBOOK_TEST_SCOPE to constrain scope).
Global default "off" (set in preview.ts).

Workflows

Local development

A11y tests are off by default globally. Opt in per story or per file with the helpers below, then run Storybook normally and check the A11y panel in the sidebar:

CI — Storybook Vitest

Full audit — ui-core inventory

Runs axe across all ui-core stories in "error" mode and produces a human-readable report plus a Datadog metrics CSV:

Internally this:

Runs Vitest with STORYBOOK_A11Y_AUDIT=1 STORYBOOK_TEST_SCOPE=ui-core --reporter=json --outputFile=a11y-inventory/vitest-results.json.
Writes raw results to a11y-inventory/vitest-results.json.

CI — merge_group inventory

Runs automatically via .github/workflows/storybook-a11y-inventory.yaml:

Trigger: merge_group (only when turbo/packages/ui-core/src/** changed), or manual workflow_dispatch.
Non-blocking: continue-on-error: true and excluded from merge-gatekeeper's required checks — a failure here never blocks a merge.
What it does: a11y:inventory:ui-core → a11y:format-datadog-metrics → uploads datadog-metrics.csv to Datadog via int128/send-datadog-action.

Chromatic

Runs on relevant non-draft PRs via .github/workflows/chromatic-storybook-ci.yaml. Opt stories in with WithSnapshot. Local publishes: pnpm chromatic.

Visual regression: captures opted-in story snapshots and blocks the PR until a reviewer accepts visual diffs. TurboSnap (onlyChanged: true) only re-snapshots affected stories; a full run is forced when design-token SCSS (packages/ui-core/src/styles/) or apps/storybook/.storybook/globals.scss changes. Snapshots auto-accept on trunk.
Accessibility tests: the build's Accessibility tab reports axe violations for snapshotted stories. The required GitHub status check is the Actions aggregate named Chromatic, not Chromatic's external UI Review status.

Datadog dashboard

Metrics are gauges — each run overwrites, so widgets use max/last per bucket, never sum.
Data is sparse by design: it only lands on merge_group runs that touch turbo/packages/ui-core/src/**.
These ui-core-only counts intentionally differ from Chromatic's totals — Chromatic spans all snapshotted ui-* packages across light and dark modes.

Metrics

Namespace: prepared.ui_core.a11y (all gauges).

Metric	Extra tags	Description
`violations.total`	—	Total axe violation count across ui-core stories
`failed_stories`	—	Stories with at least one violation
`passed_stories`	—	Stories that passed axe cleanly
`unattributed_failures`	—	Stories that failed for non-a11y reasons (axe never ran)
`errored_stories`	—	Stories where the axe runner itself errored
`violations_by_rule`	`rule:<axe-rule-id>`	Count per axe rule (e.g. `color-contrast`, `button-name`)
`violations_by_component`	`component:<folder>`, `component_path:<path>`	Count per ui-core component folder
`violations_by_impact`	`impact:critical\|serious\|moderate\|minor`	Count per axe impact level

Every row is also tagged with environment:github-actions, service:storybook-a11y, package:ui-core, repository:prepared911/prepared911, plus pr:, sha:, and ref: per run.

Note: ref currently ships empty; filter and group by pr/sha instead.

Output files

All audit output is written to turbo/apps/storybook/a11y-inventory/ (gitignored):

File	Produced by	Contents
`vitest-results.json`	`a11y:inventory:ui-core`	Raw Vitest JSON output
`summary.json`	`format-a11y-inventory.mjs`	Parsed counts: total, by rule, by component, by impact
`report.md`	`format-a11y-inventory.mjs`	Human-readable tables — open this to review findings
`datadog-metrics.csv`	`format-datadog-metrics.mjs`	CSV for `int128/send-datadog-action`

Opt-in helpers

Located in turbo/packages/util-storybook-helpers/src/with-a11y-testing.ts, imported from @prepared911/util-storybook-helpers:

Mode summary:

Mode	Storybook UI	Vitest CLI	CI
`"off"`	No a11y results	Passes	No effect
`"todo"`	Violations shown as warnings	Passes	Never blocks
`"error"`	Violations shown as errors	Fails	Blocks unless `STORYBOOK_A11Y_TODO` is set

STORYBOOK_A11Y_TODO=1 is currently set in CI so no story blocks PRs, giving the team time to fix existing violations incrementally. Per component:

Run pnpm a11y:inventory:ui-core and open a11y-inventory/report.md to find the component's violations.
Fix the violations in the component or its stories.
Upgrade the story from withA11yTesting → withA11yTestingError to lock in the clean state and get CI protection going forward.
Repeat until every ui-core component is on withA11yTestingError.

Once all ui-core stories are on "error" mode, remove STORYBOOK_A11Y_TODO from test:storybook:ci to fully gate CI. Watch the Datadog dashboard to confirm the trend is falling.

Troubleshooting

Stale results / cached bundle — clear the Vite dep cache: pnpm storybook:clean (rm -rf node_modules/.vite), then re-run.
Audit reports 0 violations on a first run — treat a suspiciously clean result as a signal to inspect the raw vitest-results.json, not a result to trust. Violations in "error" mode throw before task.meta.reports is populated, so the parser recovers them from Vitest failure messages; a parser regression can silently under-count.
A story fails for a non-a11y reason — it shows up under unattributed_failures, not the violation counts. Check the story's play function.

Testing strategy / Testing Feature Flags

CI/CD & deployment / CI/CD Overview

On this page

CI — Storybook Vitest

Full audit — ui-core inventory

CI — merge_group inventory

Burn-down (a11y debt)

Troubleshooting