This page explains the baseline testing system used in the TypeScript compiler repository: how test output is captured as baseline files, how those files are stored and versioned, how to accept new baselines, and how CI enforces baseline correctness.
For background on how tests are written and which test categories exist, see page 12.2 For the CI pipeline that validates baselines on every push, see page 13.1
Baselines are plain-text snapshot files that record the expected output of compiler tests. When a compiler test runs, it compiles TypeScript source and writes the resulting output to a local baseline file. That local file is then compared against a reference file that is checked into the repository. If they differ, the test fails.
This approach makes it easy to see exactly what changed when a compiler modification affects output: the diff between local and reference baselines shows the precise effect.
Each compiler test case can produce several baseline files, depending on what the test exercises:
| Extension | Contents |
|---|---|
.js | Emitted JavaScript output |
.d.ts | Emitted declaration files |
.js.map | Emitted source maps |
.types | Inferred type of every expression, one per line |
.symbols | Symbol resolution results for every identifier |
.errors.txt | Compiler diagnostic messages |
For example, tests/baselines/reference/libReferenceDeclarationEmit.symbols captures symbol resolution results and tests/baselines/reference/libReferenceDeclarationEmitBundle.types captures type information.
tests/baselines/
├── local/ ← Written by test runner; git-ignored
└── reference/ ← Checked into git; source of truth
The canonical path constants are defined in scripts/build/tests.mjs22-24:
localBaseline = "tests/baselines/local/" — tests write output hererefBaseline = "tests/baselines/reference/" — accepted baselines live hereThe tests/baselines/local/ directory (and several subdirectories like tests/baselines/rwc/) are listed in .gitignore11-18 and are never committed. The tests/baselines/reference/ directory is committed, with a few exclusions:
| Gitignored path | Reason |
|---|---|
tests/baselines/reference/projectOutput/* | Large generated project outputs |
tests/baselines/reference/testresults.tap | TAP report, regenerated each run |
tests/baselines/local/* | All local outputs (never committed) |
Sources: .gitignore11-21 scripts/build/tests.mjs22-24
Baseline flow diagram: test run to reference
Sources: scripts/build/tests.mjs34-193 src/testRunner/runner.ts25-48
Before each test run, cleanTestDirs() scripts/build/tests.mjs190-193 removes the entire tests/baselines/local/ directory and recreates it empty. This ensures no stale outputs from previous runs pollute comparisons.
The runConsoleTests function scripts/build/tests.mjs34-188 orchestrates the full test execution, including cleaning directories, writing a test.config file, launching Mocha, and optionally running coverage.
Test runners (CompilerBaselineRunner, FourSlashRunner, TranspileRunner, defined in src/testRunner/runner.ts58-74) each enumerate their test files, run them, and produce baseline output through the harness.
When test output changes intentionally (e.g., after fixing a bug or adding a feature), the new local output must be promoted to the reference directory. This is done with:
The baseline-accept task (defined in Herebyfile.mjs) copies files from tests/baselines/local/ to tests/baselines/reference/, replacing the old reference files. After running it, the changed reference files should be committed to the repository.
The automated maintenance workflow also uses this command. In .github/workflows/accept-baselines-fix-lints.yaml28-40 the bot:
tests/baselines/reference/ entirely via git rmnpx hereby runtests-parallel --ci --fixnpx hereby baseline-acceptThe CI pipeline includes a dedicated baselines job in .github/workflows/ci.yml368-414 Its process is deliberately strict: it regenerates all baselines from scratch, then checks that the result matches what is in the repository.
CI baseline validation flow
Sources: .github/workflows/ci.yml368-414
The three diff categories the CI checks for are:
| git diff-filter | Meaning |
|---|---|
ACR (Added/Copied/Renamed) | Tests that have no reference baseline yet |
MTUXB (Modified/Type-changed/Unmerged/Unknown/Broken) | Tests whose output changed |
D (Deleted) | Reference baselines with no corresponding test (stale files) |
If any of these categories have files, the job creates a fix_baselines.patch artifact. Contributors can download this patch and apply it locally with git apply fix_baselines.patch to see or apply the expected changes.
The regular test job also prints a baseline diff on failure .github/workflows/ci.yml125-131:
This makes it easy to diagnose which baselines changed when a normal test run fails.
Sources: .github/workflows/ci.yml125-131 .github/workflows/ci.yml368-414
Baseline system components and their relationships
Sources: scripts/build/tests.mjs22-24 src/testRunner/runner.ts58-74 Herebyfile.mjs1-50
When a test fails due to a baseline mismatch, the workflow is:
Run tests to generate local baselines:
Inspect the diff between local and reference outputs. You can set the DIFF environment variable to your preferred diff tool:
The getDiffTool() helper in scripts/build/utils.mjs155-162 reads this variable.
If the change is intentional, accept the new baselines:
Commit the updated reference files:
If the change is not intentional, investigate the compiler or test code that caused the regression before accepting.
To regenerate baselines for only one test file (e.g., myTest.ts in tests/cases/conformance/):
This limits the test runner to matching tests, minimizing how many baselines are regenerated locally.
Sources: scripts/build/options.mjs1-91 scripts/build/utils.mjs155-162
The accept-baselines-fix-lints workflow (.github/workflows/accept-baselines-fix-lints.yaml) can be triggered manually via workflow_dispatch. It:
git rm -r --quiet tests/baselines/reference--fix for lint auto-fixes)npx hereby baseline-acceptnpx hereby format"Update Baselines, Applied Lint Fixes, and/or Formatted"This is the mechanism used to batch-accept baselines after large sweeping changes to compiler output.
Sources: .github/workflows/accept-baselines-fix-lints.yaml28-40
Refresh this wiki
This wiki was recently refreshed. Please wait 4 days to refresh again.