TDM Run Results – Managing TDM Scenario Runs as Code

The Problem

A 14-scenario non-motorized sensitivity study took over a month to complete
Not because the model runs were slow — because of everything around them:
- manually editing Control Center files for each scenario
- tracking which model version ran which scenario
- reassembling scattered outputs into a writeup after the fact
This doesn’t scale. More scenarios, more analysts, more studies — same manual overhead every time.

Why We’re Sharing This

Not asking for anything today — this is what we built, what it’s bought us, and what it cost
If a similar pattern would help your agency’s scenario management, happy to talk specifics
The repo, architecture decisions (ADRs), and both example run sets are there to look at

What This Is

A framework that manages run sets — scenarios, sensitivity tests, alternatives — as versioned configuration, sitting around the TDM.

It does not touch the TDM codebase, the Cube Voyager engine, or how the model runs internally
It manages: what changes between scenarios, what got run against which model version, and what came out
Built against WFRC’s Cube Voyager TDM — the general pattern isn’t Cube-specific, the implementation details are

Workflow: From Idea to Report

flowchart LR
    A[Brainstorm concept] --> B[Prepare scoping document]
    B --> C{Review and revise}
    C -->|revise| B
    C -->|approved| D[Prepare scenario YAMLs<br>declare needed outputs]
    D --> E[Prepare custom code<br>if needed]
    E --> F[Run models, gather outputs]
    F --> G[Prepare report]

Core Idea: One Override Mechanism

Every scenario is a baseline Control Center file plus layered overrides — and input file selection and sensitivity knobs are the same kind of thing: just keys in a flat config.

# run_sets/bring-work-trips-closer-to-home/scenarios/Closer01.yaml
scenario_id: Closer01
description: "Closer01 - 10% HBW trip redistribution by City Area"
driver_script: hail-mary/__HailMary_1Subfolder_closer.s
start_from_copy: Closer00

overrides:
  vizToolDir: '...\.vizTool'
  RunDescription: "Closer01 - City Area 10% HBW trip redistribution"

Every override key is validated against the baseline before the model is touched — a typo fails in seconds, not after a multi-hour run.

How the Pieces Relate

This framework repo — config, orchestration code, curated outputs, run metadata, reports
The TDM — a separate repo, connected as a git submodule, checked out to whatever version a run set declares
Cube Voyager runs in place inside its own checkout — no forking, no parallel copies, no modification
One fixed batch entry point per TDM version; the framework doesn’t know or care what happens inside a model run, only that it starts and finishes

Running a Scenario

Validate every config/override key against the baseline
Resolve and check out the requested TDM git tag — refuses on a dirty working tree
Render the Control Center file: baseline → run-set overrides → scenario overrides → computed identity fields
Invoke the TDM’s batch entry point with the control file and scenario folder
Curate outputs — glob-selected subset only, hard per-file size ceiling, raw outputs never enter git
Record — write run_metadata.json, the source of truth for everything downstream

The Record

{
  "run_set_id": "bring-work-trips-closer-to-home",
  "scenario_id": "Closer01",
  "run_id": "20260710-133516-924e",
  "status": "success",
  "execution_mode": "manual",
  "tdm": {
    "requested_ref": "v10.0.0-beta.1-closer-to-home",
    "resolved_commit": "1fc8a350...",
    "dirty": false
  },
  "outputs": {
    "inventory_count": 1182,
    "inventory_total_bytes": 39352729618,
    "curated_count": 8
  }
}

Every run — CLI-driven or manually executed — gets one of these. Committed, schema-validated, checksummed. That’s the whole audit trail.

From Metadata to a Published Site

reports/ is a Quarto website living in the same repo — it reads run_metadata.json directly, no database
Run sets without a custom report auto-discover onto the index; non-motorized-2023 and bring-work-trips-closer-to-home have hand-built pages (slide decks + detailed writeups) reading the same underlying data
A GitHub Actions workflow renders the site on push and deploys it to GitHub Pages — publishing is automatic, not a manual export step
CI is scoped to validation and reporting only — it never touches Cube Voyager or the TDM submodule

Let’s Look at It Live

tdmruns status → GitHub Pages report site

Proof Points

non-motorized-2023 — the study that started this. 13 scenarios, run manually (block-format parsing wasn’t solved yet), outputs curated and reported through the framework. Since retired: outputs purged, reports now read a small frozen snapshot instead of ~500 MB of curated CSVs.

bring-work-trips-closer-to-home — currently running. 9 scenarios (3 geography types × 3 shift levels) plus baseline. As of this week, scenarios are executing through the real CLI end-to-end (bin/RunModel.bat → Cube Voyager), not just manual import — the Control Center block-format blocker that held back the first study is now solved for this one.

What’s Not Solved Yet

The Cube block-format parser was genuinely hard — Cube Voyager’s native KEY = value Control Center format, not YAML. Real engineering effort, not a config change.
Exit-code handling is imperfect — Voyager’s cluster-shutdown step sometimes returns nonzero after a model finished successfully; runs get marked “failed” that actually aren’t yet.
Cube’s per-machine licensing caps throughput — this doesn’t parallelize execution across machines.
Two run sets have exercised this end-to-end so far. Real, but not battle-tested.
Requires git fluency from analysts — a real onboarding cost.

What’s Cube-Specific vs. General

WFRC/Cube-specific	Generalizes to any TDM
Native `.block` Control Center parsing	Versioned run configs as code
Fixed batch entry point, driver-script staging	One override mechanism, validated before execution
`Scenarios/_default/` baseline library convention	Curated outputs + size ceiling, raw outputs stay local
Cube’s exit-code/cluster-shutdown quirks	Structured run metadata as the audit trail
	Automated reporting from that metadata

Questions

Repo: github.com/WFRCAnalytics/WF-TDM-Runs Live reports: wfrcanalytics.github.io/WF-TDM-Runs