Validation

Real enrollment vs. PopulationSim synthetic estimates, by district and county

Overview

This is the page the site is built around: where the real enrollment numbers (2_enrollment.qmd) and PopulationSim’s synthetic estimates (3_synthetic.qmd) get compared.

• Objective: for each school district, compare the synthetic enrolled count (school grades) and the synthetic population aged 3-18 against real enrollment from Utah DOE or NCES CCD. For each county, compare the same kind of synthetic counts against USHE or IPEDS. The population counts are a sanity bound, since age is controlled in PopulationSim, so they are not expected to match.
• Using the page: the inputs above each table pick what is compared and against which source. The table and the scatter plots update with them. The flag marks rows that are further off than the threshold you set.
• Data vintages: Utah DOE 2024-25 (the BY2023 UGRC snapshot, SchoolYear 2025), NCES CCD and PSS 2023-24, USHE Academic Year 2025 Term 2, IPEDS fall 2023, and PopulationSim 2023. Every chart and table names the vintage next to each source.
• What it reads: the district and county tables from 2_enrollment.qmd and the synthetic tables from 3_synthetic.qmd, all in _data/intermediate/.
• Status: sections 1 and 2 (districts and counties) are built. Section 3 (Box Elder) and the summary in section 4 are still placeholders.
• Box Elder: reported separately, since its real and synthetic numbers are both scoped differently there (model region vs. full county). It is still a TODO.

// Tooltip colours follow the page theme. Plot draws each tooltip box with a fixed background and its
// text inherits the page colour, so the box colour has to match the theme. This re-runs when the theme
// changes (the observers below), so the charts redraw with the new tooltip colours.
tipTheme = Generators.observe(notify => {
  const backgroundOf = el => {
    const bg = getComputedStyle(el).backgroundColor;
    return bg.startsWith("rgba(0, 0, 0, 0)") || bg === "transparent" ? null : bg;
  };
  const isDark = () => {
    const bg = backgroundOf(document.body) ?? backgroundOf(document.documentElement) ?? "rgb(255, 255, 255)";
    const [r, g, b] = bg.match(/[\d.]+/g).slice(0, 3).map(Number);
    return 0.299 * r + 0.587 * g + 0.114 * b < 128;
  };
  const emit = () => notify(isDark()
    ? {fill: "#0c2a3b", stroke: "#4b5563"}
    : {fill: "#ffffff", stroke: "#374151"});
  const observer = new MutationObserver(emit);
  observer.observe(document.documentElement, {attributes: true, attributeFilter: ["class", "style", "data-bs-theme", "data-theme"]});
  observer.observe(document.body, {attributes: true, attributeFilter: ["class", "style"]});
  observer.observe(document.head, {subtree: true, attributes: true, attributeFilter: ["href", "disabled", "media"]});
  const media = window.matchMedia("(prefers-color-scheme: dark)");
  media.addEventListener("change", emit);
  emit();
  return () => {
    observer.disconnect();
    media.removeEventListener("change", emit);
  };
})

1. School validation (by district)

compareOptions = ["Enrolled (school grades)", "Population aged 3-18"];
referenceOptions = ["Utah DOE 2024-25", "NCES CCD 2023-24"];
measureOptions = ["Difference", "Percent difference", "Ratio"];

districtReal = FileAttachment("_data/intermediate/real_enrollment_by_district.csv").csv({typed: true});
districtSynth = FileAttachment("_data/intermediate/synthetic_k12_by_district.csv").csv({typed: true});
districtSynthMap = new Map(districtSynth.map(d => [d["District Name"], d]));

districtRefKey = `Reference (${districtReference})`;
districtSynKey = districtCompare === compareOptions[0] ? "Synthetic enrolled (PopSim 2023)" : "Synthetic population (PopSim 2023)";

viewof districtCompare = Inputs.select(compareOptions, {label: "Compare", value: compareOptions[0]})

viewof districtReference = Inputs.select(referenceOptions, {label: "Reference", value: referenceOptions[0]})

viewof districtMeasure = Inputs.select(measureOptions, {label: "Show", value: measureOptions[1]})

viewof districtFlag = Inputs.range([0, 50], {label: "Flag if off by more than (%)", step: 1, value: 10})

Table

districtTable = districtReal.map(d => {
  const s = districtSynthMap.get(d["District Name"]);
  const ref = districtReference === referenceOptions[0] ? d.doe_enrollment : d.nces_enrollment;
  const syn = s ? (districtCompare === compareOptions[0] ? s.sch_grade : s.age_3_18) : null;
  const diff = syn != null && ref != null ? syn - ref : null;
  const pct = diff != null && ref ? diff / ref : null;
  const ratio = diff != null && ref ? syn / ref : null;
  const shown = districtMeasure === "Difference" ? diff : districtMeasure === "Percent difference" ? pct : ratio;
  return {
    District: d["District Name"],
    [districtRefKey]: ref,
    [districtSynKey]: syn,
    [districtMeasure]: shown,
    Flag: pct != null && Math.abs(pct) * 100 > districtFlag ? "⚠" : ""
  };
})

countFormat = v => v == null ? "—" : Math.round(v).toLocaleString("en-US");
shownFormat = districtMeasure === "Percent difference"
  ? (v => v == null ? "—" : `${(v * 100).toFixed(1)}%`)
  : districtMeasure === "Ratio"
    ? (v => v == null ? "—" : v.toFixed(2))
    : (v => v == null ? "—" : `${v > 0 ? "+" : ""}${Math.round(v).toLocaleString("en-US")}`);

Inputs.table(districtTable, {
  columns: ["District", districtRefKey, districtSynKey, districtMeasure, "Flag"],
  format: {[districtRefKey]: countFormat, [districtSynKey]: countFormat, [districtMeasure]: shownFormat},
  maxHeight: "none"
})

Table 1: Real and synthetic students by district. Utah DOE is 2024-25, NCES CCD is 2023-24 and the synthetic population is PopulationSim 2023. Percent difference is (synthetic minus reference) divided by reference. A row is flagged when its percent difference is larger than the threshold.

(a)

(b)

(c)

(d)

Scatter

scatterRows = districtReal.flatMap(d => {
  const s = districtSynthMap.get(d["District Name"]);
  if (!s) return [];
  const name = d["District Name"];
  return [
    {reference: referenceOptions[0], measure: "Enrolled", x: d.doe_enrollment, y: s.sch_grade, district: name},
    {reference: referenceOptions[0], measure: "Population", x: d.doe_enrollment, y: s.age_3_18, district: name},
    {reference: referenceOptions[1], measure: "Enrolled", x: d.nces_enrollment, y: s.sch_grade, district: name},
    {reference: referenceOptions[1], measure: "Population", x: d.nces_enrollment, y: s.age_3_18, district: name}
  ];
});

scatterMax = d3.max(scatterRows, d => Math.max(d.x, d.y));

diagRows = referenceOptions.flatMap(reference =>
  ["Enrolled", "Population"].flatMap(measure => [
    {reference, measure, x: 0, y: 0},
    {reference, measure, x: scatterMax, y: scatterMax}
  ])
);

Plot.plot({
  width: 760,
  height: 640,
  grid: true,
  style: {background: "transparent", color: "inherit"},
  fx: {label: null},
  fy: {label: null},
  x: {label: "Reference (students)", grid: true},
  y: {label: "Synthetic (persons)", grid: true},
  marks: [
    Plot.line(diagRows, {x: "x", y: "y", fx: "measure", fy: "reference", stroke: "#999", strokeDasharray: "4 4"}),
    Plot.dot(scatterRows, {
      x: "x", y: "y", fx: "measure", fy: "reference",
      fill: "#1f78b4", r: 4, tip: tipTheme, title: d => d.district
    })
  ]
})

(a)

(b)

(c)

(d)

Figure 1: Synthetic (PopulationSim 2023) against real (Utah DOE 2024-25, NCES CCD 2023-24), by district, for each reference source and each synthetic measure. Points on the dashed line agree exactly. Each panel uses the same scale.

Enrolled by school level

Pick any combination of school levels and districts. The chart adds up the selection: each district shows real DOE and NCES enrollment next to the synthetic grade count and the synthetic age-band count.

levelOrder = ["Pre-K", "K", "Elementary", "Middle", "High"];
levelSources = ["DOE 2024-25", "NCES 2023-24", "Synthetic grade (PopSim 2023)", "Synthetic age band (PopSim 2023)"];

// Age bands used for the sanity bound: one band per level, no overlaps, so each age belongs to one level
ageBands = ({"Pre-K": [3, 4], "K": [5, 5], "Elementary": [6, 11], "Middle": [12, 14], "High": [15, 18]});

realByLevel = FileAttachment("_data/intermediate/real_by_district_level.csv").csv({typed: true});
synthByLevel = FileAttachment("_data/intermediate/synthetic_by_district_level.csv").csv({typed: true});
synthAge = FileAttachment("_data/intermediate/synthetic_age_by_district.csv").csv({typed: true});

levelRows = realByLevel.map(d => {
  const s = synthByLevel.find(x => x["District Name"] === d["District Name"] && x.Level === d.Level);
  return {
    district: d["District Name"],
    level: d.Level,
    doe: d.doe_enrollment ?? 0,
    nces: d.nces_enrollment ?? 0,
    grade: s ? s.sch_grade : 0
  };
});

districtNames = [...new Set(levelRows.map(d => d.district))].sort();

viewof selectedLevels = Inputs.select(levelOrder, {
  label: "School levels",
  multiple: 5,
  value: levelOrder,
  format: lvl => ageBands[lvl][0] === ageBands[lvl][1]
    ? `${lvl} (Age: ${ageBands[lvl][0]})`
    : `${lvl} (Age: ${ageBands[lvl][0]}-${ageBands[lvl][1]})`
})

viewof selectedDistricts = Inputs.select(districtNames, {
  label: "Districts",
  multiple: 5,
  value: districtNames
})

viewof splitByDistrict = Inputs.checkbox(["Split by district"], {
  label: "Chart",
  value: ["Split by district"]
})

selectedFacet = splitByDistrict.length > 0

chartRows = {
  const out = [];
  for (const dist of selectedDistricts) {
    const rows = levelRows.filter(d => d.district === dist && selectedLevels.includes(d.level));
    if (!rows.length) continue;
    const ages = new Set();
    for (const lvl of selectedLevels) {
      const [lo, hi] = ageBands[lvl];
      for (let a = lo; a <= hi; a++) ages.add(a);
    }
    const ageTotal = d3.sum(
      synthAge.filter(d => d["District Name"] === dist && ages.has(d.age)),
      d => d.persons
    );
    out.push({district: dist, source: levelSources[0], students: d3.sum(rows, d => d.doe)});
    out.push({district: dist, source: levelSources[1], students: d3.sum(rows, d => d.nces)});
    out.push({district: dist, source: levelSources[2], students: d3.sum(rows, d => d.grade)});
    out.push({district: dist, source: levelSources[3], students: ageTotal});
  }
  if (selectedFacet) return out;
  // Split off: add the selected districts into one total per source
  return d3.rollups(out, v => d3.sum(v, d => d.students), d => d.source)
    .map(([source, students]) => ({district: "Total", source, students}));
}

chartRows.length === 0
  ? html`<p><em>Select at least one school level and one district.</em></p>`
  : Plot.plot({
      width: 960,
      height: 460,
      marginBottom: 60,
      grid: true,
      style: {background: "transparent", color: "inherit"},
      ...(selectedFacet ? {fx: {label: null, domain: selectedDistricts}} : {}),
      x: {axis: null},
      y: {label: "Students", grid: true},
      color: {
        legend: true,
        domain: levelSources,
        range: ["#1f78b4", "#33a02c", "#e31a1c", "#ff7f00"]
      },
      marks: [
        Plot.barY(chartRows, {
          ...(selectedFacet ? {fx: "district"} : {}),
          x: "source",
          y: "students",
          fill: "source",
          tip: tipTheme
        }),
        Plot.ruleY([0])
      ]
    })

(a)

(b)

Figure 2: Students by district for the selected school levels. DOE (2024-25) and NCES (2023-24) are K-12 plus preschool enrollment. Synthetic counts are PopulationSim 2023. Synthetic grade counts persons in the same grades. Synthetic age band counts persons in the selected levels’ age bands, with each person counted once.

2. College validation (by county)

collegeCompareOptions = ["Enrolled (undergraduate and graduate)", "Population aged 18-24"];
collegeReferenceOptions = ["USHE AY 2025 (public, all levels)", "IPEDS fall 2023 (campus-based)"];

realCollegeLevel = FileAttachment("_data/intermediate/real_college_by_county_level.csv").csv({typed: true});
realCountyTotals = FileAttachment("_data/intermediate/real_enrollment_by_county.csv").csv({typed: true});
synthCollege = FileAttachment("_data/intermediate/synthetic_college_by_county.csv").csv({typed: true});

countyOrder = [3, 11, 35, 49, 57];
countyName = ({3: "Box Elder", 11: "Davis", 35: "Salt Lake", 49: "Utah", 57: "Weber"});

collegeRefKey = `Reference (${collegeReference.startsWith("USHE") ? "USHE AY 2025" : ipedsLabel})`;
collegeSynKey = collegeCompare === collegeCompareOptions[0] ? "Synthetic enrolled (PopSim 2023)" : "Synthetic age 18-24 (PopSim 2023)";

viewof collegeCompare = Inputs.select(collegeCompareOptions, {label: "Compare", value: collegeCompareOptions[0]})

viewof collegeReference = Inputs.select(collegeReferenceOptions, {label: "Reference", value: collegeReferenceOptions[0]})

viewof collegeMeasure = Inputs.select(measureOptions, {label: "Show", value: measureOptions[1]})

viewof collegeFlag = Inputs.range([0, 50], {label: "Flag if off by more than (%)", step: 1, value: 10})

Table

collegeTable = countyOrder.map(c => {
  const syn = synthCollege.find(d => d.COUNTYFP === c);
  const ushe = realCountyTotals.find(d => d.COUNTYFP === c)?.ushe_enrollment ?? null;
  const ipeds = d3.sum(realCollegeLevel.filter(d => d.COUNTYFP === c && ipedsInScope(d)), d => d.nces_enrollment);
  const ref = collegeReference.startsWith("USHE") ? ushe : ipeds;
  const enrolled = collegeCompare === collegeCompareOptions[0];
  const synVal = syn ? (enrolled ? syn.undergraduate + syn.graduate : syn.age_18_24) : null;
  const diff = synVal != null && ref != null ? synVal - ref : null;
  const pct = diff != null && ref ? diff / ref : null;
  const ratio = diff != null && ref ? synVal / ref : null;
  const shown = collegeMeasure === "Difference" ? diff : collegeMeasure === "Percent difference" ? pct : ratio;
  return {
    County: countyName[c],
    [collegeRefKey]: ref,
    [collegeSynKey]: synVal,
    [collegeMeasure]: shown,
    Flag: pct != null && Math.abs(pct) * 100 > collegeFlag ? "⚠" : ""
  };
})

Inputs.table(collegeTable, {
  columns: ["County", collegeRefKey, collegeSynKey, collegeMeasure, "Flag"],
  format: {
    [collegeRefKey]: countFormat,
    [collegeSynKey]: countFormat,
    [collegeMeasure]: collegeMeasure === "Percent difference"
      ? (v => v == null ? "—" : `${(v * 100).toFixed(1)}%`)
      : collegeMeasure === "Ratio"
        ? (v => v == null ? "—" : v.toFixed(2))
        : (v => v == null ? "—" : `${v > 0 ? "+" : ""}${Math.round(v).toLocaleString("en-US")}`)
  }
})

Table 2: Real and synthetic college students by county. USHE (Academic Year 2025 Term 2) covers public institutions only, all levels. IPEDS (fall 2023) is campus-based enrollment (in-person plus hybrid). Synthetic enrolled (PopulationSim 2023) counts residents at any college. Percent difference is (synthetic minus reference) divided by reference.

(a)

(b)

Scatter

collegeScatterRows = countyOrder.flatMap(c => {
  const syn = synthCollege.find(d => d.COUNTYFP === c);
  if (!syn) return [];
  const ushe = realCountyTotals.find(d => d.COUNTYFP === c)?.ushe_enrollment;
  const ipeds = d3.sum(realCollegeLevel.filter(d => d.COUNTYFP === c && ipedsInScope(d)), d => d.nces_enrollment);
  const enrolledSyn = syn.undergraduate + syn.graduate;
  const name = countyName[c];
  return [
    {reference: "USHE AY 2025", measure: "Enrolled", x: ushe, y: enrolledSyn, county: name},
    {reference: "USHE AY 2025", measure: "Population", x: ushe, y: syn.age_18_24, county: name},
    {reference: "IPEDS fall 2023", measure: "Enrolled", x: ipeds, y: enrolledSyn, county: name},
    {reference: "IPEDS fall 2023", measure: "Population", x: ipeds, y: syn.age_18_24, county: name}
  ].filter(d => d.x != null);
});

collegeScatterMax = d3.max(collegeScatterRows, d => Math.max(d.x, d.y));

collegeDiagRows = ["USHE AY 2025", "IPEDS fall 2023"].flatMap(reference =>
  ["Enrolled", "Population"].flatMap(measure => [
    {reference, measure, x: 0, y: 0},
    {reference, measure, x: collegeScatterMax, y: collegeScatterMax}
  ])
);

Plot.plot({
  width: 760,
  height: 640,
  grid: true,
  style: {background: "transparent", color: "inherit"},
  fx: {label: null},
  fy: {label: null},
  x: {label: "Reference (students)", grid: true},
  y: {label: "Synthetic (persons)", grid: true},
  marks: [
    Plot.line(collegeDiagRows, {x: "x", y: "y", fx: "measure", fy: "reference", stroke: "#999", strokeDasharray: "4 4"}),
    Plot.dot(collegeScatterRows, {
      x: "x", y: "y", fx: "measure", fy: "reference",
      fill: "#1f78b4", r: 5, tip: tipTheme, title: d => d.county
    })
  ]
})

(a)

(b)

(c)

(d)

Figure 3: Synthetic (PopulationSim 2023) against real (USHE AY 2025, IPEDS fall 2023), by county, for each reference source and each synthetic measure. Points on the dashed line agree exactly. Each panel uses the same scale.

Colleges by county

Pick the counties and college levels to compare. USHE covers public institutions only and has no level split, so its bar always shows all levels. The age band counts persons aged 18-24 whatever the level, as a sanity bound.

viewof selectedCounties = Inputs.select(countyOrder, {
  label: "Counties",
  multiple: 5,
  value: countyOrder,
  format: c => countyName[c]
})

viewof selectedCollegeLevels = Inputs.select(["Undergraduate", "Graduate"], {
  label: "College level",
  multiple: 2,
  value: ["Undergraduate", "Graduate"]
})

viewof collegeIncludePrivate = Inputs.checkbox(["Include private institutions"], {
  label: "Institutions",
  value: ["Include private institutions"]
})

viewof splitByCounty = Inputs.checkbox(["Split by county"], {
  label: "Chart",
  value: ["Split by county"]
})

selectedCountySplit = splitByCounty.length > 0;
includePrivate = collegeIncludePrivate.length > 0;
ipedsLabel = includePrivate ? "IPEDS fall 2023, all institutions" : "IPEDS fall 2023, public only";
ipedsInScope = d => includePrivate || d.Control === "Public";

collegeRows = {
  const out = [];
  for (const c of selectedCounties) {
    const ipeds = d3.sum(
      realCollegeLevel.filter(d => d.COUNTYFP === c && selectedCollegeLevels.includes(d.Level) && ipedsInScope(d)),
      d => d.nces_enrollment
    );
    const ushe = realCountyTotals.find(d => d.COUNTYFP === c)?.ushe_enrollment ?? 0;
    const syn = synthCollege.find(d => d.COUNTYFP === c) ?? {undergraduate: 0, graduate: 0, age_18_24: 0};
    const synEnrolled =
      (selectedCollegeLevels.includes("Undergraduate") ? syn.undergraduate : 0) +
      (selectedCollegeLevels.includes("Graduate") ? syn.graduate : 0);
    out.push({county: countyName[c], source: "USHE AY 2025 (all levels)", students: ushe});
    out.push({county: countyName[c], source: ipedsLabel, students: ipeds});
    out.push({county: countyName[c], source: "Synthetic enrolled (PopSim 2023)", students: synEnrolled});
    out.push({county: countyName[c], source: "Synthetic age 18-24 (PopSim 2023)", students: syn.age_18_24});
  }
  if (selectedCountySplit) return out;
  // Split off: add the selected counties into one total per source
  return d3.rollups(out, v => d3.sum(v, d => d.students), d => d.source)
    .map(([source, students]) => ({county: "Total", source, students}));
}

chartCollege = collegeRows.length === 0
  ? html`<p><em>Select at least one county and one college level.</em></p>`
  : Plot.plot({
      width: 960,
      height: 460,
      marginBottom: 60,
      grid: true,
      style: {background: "transparent", color: "inherit"},
      ...(selectedCountySplit ? {fx: {label: null, domain: selectedCounties.map(c => countyName[c])}} : {}),
      x: {axis: null},
      y: {label: "Students", grid: true},
      color: {
        legend: true,
        domain: ["USHE AY 2025 (all levels)", ipedsLabel, "Synthetic enrolled (PopSim 2023)", "Synthetic age 18-24 (PopSim 2023)"],
        range: ["#1f78b4", "#33a02c", "#e31a1c", "#ff7f00"]
      },
      marks: [
        Plot.barY(collegeRows, {
          ...(selectedCountySplit ? {fx: "county"} : {}),
          x: "source",
          y: "students",
          fill: "source",
          tip: tipTheme
        }),
        Plot.ruleY([0])
      ]
    })

(a)

(b)

Figure 4: College enrollment by county for the selected levels. USHE (Academic Year 2025 Term 2) is public institutions only, all levels. IPEDS (fall 2023) is campus-based enrollment (in-person plus hybrid) at every institution in the county that IPEDS covers. Synthetic (PopulationSim 2023) counts residents enrolled at any college, by SCHG level. The age band is not split by level.

3. Box Elder

# TODO: same comparisons as sections 1-2, scoped to Box Elder only, with the model-boundary
#   clipping from 3_synthetic.qmd's Box Elder handling applied -- reported here as its own
#   section rather than folded into the main district/county tables

4. Summary