Research Design Guide

Wildfire Smoke and Transportation

Byeong-Hak Choe

SUNY Geneseo

November 2, 2026

Research design workflow

This instructor-authored guide uses public smoke and transportation data to illustrate a reproducible study.

  • Define an outcome that the data can measure.
  • Build and audit one consistent analysis unit.
  • Separate descriptive evidence, prediction, and causal interpretation.

An illustrative scope

Begin with daily flight disruptions at Northeast airports. A county-quarter analysis of transportation employment and wages can be a separate extension.

Code snippets are templates. Confirm the source schema, sample definition, and required inputs before running them.

🎯 Research Question

Outcomes and the research question

Activity measures

  • Flight cancellations and delays, daily
  • Transit ridership, monthly
  • Transportation employment and wages, quarterly
  • Border crossings, monthly

Limits on interpretation

These outcomes measure activity, service disruptions, or labor-market conditions. They do not directly measure firm revenue or profit.

An illustrative question

How does exposure to Canadian wildfire smoke relate to flight cancellations and delays at Northeast airports?

A testable question needs an estimand

Define the comparison before choosing an estimator:

Element Example decision
Unit One origin airport on one local calendar day
Outcome Cancelled flights divided by scheduled flights
Exposure County surface-smoke PM2.5
Comparison Within-airport variation, with a defended adjustment set
Claim A descriptive association, prediction, or causal effect

Comparisons across modes require compatible scales and observation periods.

Policy stakes: keep what the design can answer

Potential policy uses

  • Airport and DOT alert thresholds
  • The cost side of cross-border smoke policy
  • Which modes and communities most need operating support

Additional evidence needed

  • Insurance pricing requires evidence about the relevant risk and decision.
  • Cost–benefit analysis requires a defined policy comparison and measurable costs.
  • Visibility requires documentation of how operating rules and smoke conditions interact.

Connect each proposed policy use to a result that could inform a specific decision.

📚 Literature

From a list of papers to a map of the debate

Organize the literature by mechanism, outcome, and research design. The course bibliography supplies starting points:

Debate Start with Research-design use
Smoke and labor markets Borgschulte et al. (2024) A comparison for a QCEW extension
Demand channel = avoidance Burke et al. (2022); Neidell (2009) People change behavior when smoke or alerts arrive
Flight data in an environmental design Schlenker and Walker (2016) How airport operations enter a credible design
Why the Northeast may differ Heutel et al. (2021) Harm per hot day is smaller where heat is common; smoke may hurt more where it is rare

The course bibliography contains these references. Support any proposed literature gap with a documented search and study-by-study comparison.

A published labor-market design as a comparison

  • Cite the published version: Review of Economics and Statistics 106(6), 1558–1575—not the 2022 NBER working paper
  • A transportation-sector QCEW analysis can compare its exposure, outcome, and timing with the published design.
Borgschulte et al. (2024) Transportation example
Smoke HMS smoke days, nationwide Canadian-dominated smoke, Northeast
Outcome County labor-market outcomes (earnings, employment) Transportation employment and wages, NAICS 48–49
Timing Smoke days per quarter Match it, so the two estimates can be compared

Read the published data and specification sections before constructing a comparable panel.

Sample windows and extreme seasons

2006200720082009 2010201120122013 2014201520162017 201820192020 20212022 202320242025
Illustrative 2018–2025 window COVID window (excluded) Heavy Canadian-dominated smoke in the Mid-Atlantic (examples)

Short window, COVID period removed

5 years

2018, 2019, 2023, 2024, and 2025 remain. Check how concentrated exposure is in the extreme smoke seasons.

2006–2025, COVID removed

17 years

BTS on-time data, QCEW, and the smoke data all cover the full period. Canadian smoke reached the East before 2018: check 2010 in canadian_dominated_summary.csv.

Excluding March 2020–December 2022 also drops the 2021 episode. State that cost; a robustness check can bring 2021 back.

🗂️ Data

Data documentation and coverage

Record for each source

  • Provider and release version
  • Licence and access date
  • Spatial and temporal coverage
  • Observation unit, variables, units, and provenance

Check before combining sources

  • Changes in source releases or reporting rules
  • Estimates, suppression, and incomplete coverage
  • Exact table names and join keys
  • Whether a variable is observed, modeled, or derived

Use the course data guide for bundle definitions and cite the original providers.

Three data details that change results

1 · The Canadian share is a proxy

Share of same-day fire radiative power within 1,500 km that is in Canada. It ignores winds and smoke height.

Transported smoke takes days to arrive, so compare share_ca_lag03 (days \(d-3\) to \(d\)) with the same-day share. The bundle flag ca_dominated_sameday uses 50%.

2 · Sparse shares are not a complete smoke series

The county-day fire-share table includes only days with positive unrounded CSU smoke.

Read smoke from the full county-day series. A missing share row can accompany zero or negative smoke; it does not justify replacing smoke with zero. Keep unavailable fire shares as NA.

3 · Verify each product’s day definition

Check the BTS flight-date convention, the exact CSU smoke release, and the weather product used in the analysis. For ERA5, verify how hourly UTC observations were aggregated to a daily value.

A local calendar day and a UTC day cover different hours. Document the alignment rule before joining daily observations.

Reading BTS cancellation codes

Code Cause Possible design use
A Carrier —
B Weather Mechanism evidence
C National Aviation System: non-extreme weather, air-traffic-control limits Mechanism evidence: FAA flow restrictions in smoke
D Security Placebo outcome: smoke should not move it

Use all cancellations and delays as the outcome

Smoke-related cancellations need not be coded B; carriers may record them under C. Use the codes as evidence on the mechanism, not to define the outcome.

Verify the reporting-carrier threshold for each sample year: the set of carriers that report can change over time.

🔍 Intake and Audit

A reproducible intake pipeline

01_bts_download.R

Monthly PREZIP files; keep Northeast flights as you read

→

02_bts_clean.R

Airport-day panel; grain check

→

03_merge_smoke.R

Airport → county; CSU smoke, Canadian share, weather

→

04_audit.R

Rows, keys, duplicates, missing values, ranges → a table in the report

  • Keep raw files read-only; write clean files only after a grain check
  • BTS is large (several million flights a year nationally): filter while reading
  • Save the actual audit results alongside the code and input-file provenance.

Starter code: the airport-day panel

library(tidyverse)
ne_states <- c("CT", "ME", "MA", "NH", "RI", "VT", "NJ", "NY", "PA")

# One unzipped monthly PREZIP file; column names as in TranStats
read_bts_month <- function(path) {
  read_csv(path, show_col_types = FALSE,
           col_select = c(FlightDate, Origin, OriginState,
                          Cancelled, CancellationCode)) |>
    filter(OriginState %in% ne_states)
}

airport_day <- list.files("data/raw/bts", pattern = "\\.csv$",
                          full.names = TRUE) |>
  map(read_bts_month) |>
  list_rbind() |>
  summarise(n_sched  = n(),
            n_cancel = sum(Cancelled),
            n_wx     = sum(CancellationCode %in% "B"),
            n_nas    = sum(CancellationCode %in% "C"),
            n_sec    = sum(CancellationCode %in% "D"),
            .by = c(Origin, FlightDate)) |>
  mutate(cancel_rate = n_cancel / n_sched)
stopifnot(!anyDuplicated(airport_day[c("Origin", "FlightDate")]))  # one row per airport-day

%in% treats the blank cancellation code of an operated flight as “no match,” where == would return NA.

Starter code: attach smoke

library(sf)
ne_fips <- c("09", "23", "25", "33", "44", "50", "34", "36", "42")
counties <- st_read("wildfire_data/clean/counties_2020.gpkg", quiet = TRUE) |>
  filter(census_region == "Northeast")

# airports: one row per Origin with lon and lat (BTS airport master coordinates)
airports_sf <- st_as_sf(airports, coords = c("lon", "lat"), crs = 4326)
airport_county <- airports_sf |>
  st_join(select(counties, GEOID), join = st_nearest_feature) |>
  st_drop_geometry()

read_csu <- function(f) {
  read_csv(f, col_types = cols(GEOID = col_character())) |>
    filter(str_sub(GEOID, 1, 2) %in% ne_fips) |>
    select(GEOID, date, smoke_pm25)
}
csu <- list.files("wildfire_data/clean/csu_smoke_county_day", full.names = TRUE) |>
  map(read_csu) |>
  list_rbind()

panel <- airport_day |>
  left_join(airport_county, by = "Origin") |>
  left_join(csu, by = join_by(GEOID, FlightDate == date))

st_nearest_feature catches coastal airports that fall just outside the shoreline-clipped county outlines.

Report the audit as results

Create an audit table from each input actually read:

audit <- function(df, key, date_col) {
  tibble(rows      = nrow(df),
         dup_keys  = sum(duplicated(df[key])),
         rows_w_na = sum(!complete.cases(df)),
         first     = min(df[[date_col]]),
         last      = max(df[[date_col]]))
}

bind_rows(
  airport_day = audit(airport_day, c("Origin", "FlightDate"), "FlightDate"),
  csu         = audit(csu, c("GEOID", "date"), "date"),
  .id = "file"
)

COVID is a design decision, not a check

Changes in transportation demand during the pandemic can affect the comparison. Pre-specify whether and how to exclude a pandemic period, explain the rationale, and compare sensitivity to that choice.

Negative smoke is not an error

\[ \text{smoke}_{ct} = \text{HMS flag}_{ct} \times \left(\text{total PM2.5}_{ct} - \text{background}_{ct}\right) \]

  • When the plume stays aloft, ground-level PM2.5 can fall below the seasonal background, so smoke < 0
  • In 2023, 6.8% of CSU county-days are negative (1.4% in 2006 to 7.9% in 2024)
  • The data guide says to keep them: dropping or flooring them biases smoke upward
  • Keep valid negative values in the continuous smoke series. Threshold-based episode or person-day summaries answer a different question; do not apply their classification rules to every analysis.

📊 Figure and Map

Descriptive figures and adjusted comparisons

Begin with an episode plot and a clearly defined binned comparison. Removing weather or calendar effects is a modeling choice that needs explanation.

Figure A · One episode

  • June 1–15, 2023 at JFK, LGA, EWR, and PHL
  • Daily CSU smoke PM2.5 and the daily cancellation rate
  • Comparison: the same days of 2022 or 2024

If the visibility mechanism is real, it should show here first.

Figure B · All airport-days

  • Northeast airports, 2006–2025, COVID window excluded
  • Mean cancellation rate by smoke bin: ≤0, 0–5, 5–15, 15–35, >35 µg/m³
  • After removing airport-by-month and weekday means

Treat these as illustrative bins. Report the observations in each and check sensitivity to the cut points.

Starter code: Figure B

library(fixest)

fig_b <- panel |>
  filter(!between(FlightDate, as.Date("2020-03-01"), as.Date("2022-12-31"))) |>
  mutate(month     = month(FlightDate),
         wday      = wday(FlightDate, label = TRUE),
         smoke_bin = cut(smoke_pm25, c(-Inf, 0, 5, 15, 35, Inf),
                         labels = c("<= 0", "0-5", "5-15", "15-35", "> 35")))

# Remove airport-by-month and weekday means
fit0 <- feols(cancel_rate ~ 1 | Origin^month + wday, data = fig_b)
fig_b$resid <- resid(fit0, na.rm = FALSE)

fig_b |>
  filter(!is.na(smoke_bin)) |>
  summarise(mean_resid = mean(resid, na.rm = TRUE), n = n(), .by = smoke_bin) |>
  ggplot(aes(smoke_bin, mean_resid)) +
  geom_col() +
  labs(x = "Smoke PM2.5 (µg/m³)",
       y = "Cancellation rate relative to\nairport-month and weekday means")

Report n for each bin in the caption: the top bins will hold few airport-days, and readers need to know that.

An exposure map from the course bundle

Northeast counties shaded by Canadian-dominated days above 15 µg/m³ (2006–2025), with airports sized by flights. Classwork 5 covers the joins.

ca_days <- read_csv("wildfire_data/clean/county_day_fire_share_2006_2025.csv.gz",
                    col_select = c(GEOID, smoke_pm25, ca_dominated_lag03),
                    col_types = cols(GEOID = col_character())) |>
  filter(str_sub(GEOID, 1, 2) %in% ne_fips, smoke_pm25 > 15, ca_dominated_lag03) |>
  count(GEOID, name = "ca_days_gt15")

flights_by_airport <- summarise(airport_day, flights = sum(n_sched), .by = Origin)

counties |>
  left_join(ca_days, by = "GEOID") |>
  mutate(ca_days_gt15 = replace_na(ca_days_gt15, 0)) |>   # here, no row = zero days
  ggplot() +
  geom_sf(aes(fill = ca_days_gt15), color = "white", linewidth = 0.1) +
  geom_sf(data = inner_join(airports_sf, flights_by_airport, by = "Origin"),
          aes(size = flights), shape = 21, fill = "white") +
  scale_fill_viridis_c(name = "Canadian-dominated\ndays above 15 µg/m³") +
  theme_void()

📐 Design

Baseline: an airport-day panel model

\[ \text{CancelRate}_{at} = \beta\, \text{Smoke}_{c(a),t} + W_{c(a),t}'\gamma + \alpha_{a \times m(t)} + \delta_{\text{dow}(t)} + \lambda_{y(t)} + \varepsilon_{at} \]

\(a\): airport · \(t\): day · \(c(a)\): the airport’s county · \(W\): precipitation, temperature, wind · \(\alpha_{a \times m}\): airport-by-month-of-year effects · \(\delta\): day-of-week (and holiday) effects · \(\lambda\): year effects

Unit and outcome

  • Airport-day (or flight), not county-day
  • Cancellation rate, or a Poisson count with scheduled flights as exposure (fepois(), offset = ~ log(n_sched))
  • The log of a count is undefined on the many zero days

Sample and extension

  • 2006–2025, excluding March 2020–December 2022
  • Add smoke at the destination airport: it separates smoke at the origin from network delays elsewhere

Inference: cluster by airport and by date

Within an airport, over time

Lecture 7’s problem: shocks persist within a unit → cluster by airport

Across airports, on the same day

Smoke is a regional shock: on June 7, 2023, nearly every Northeast airport was treated at once → cluster by date

# panel now also holds weather (tmax, prcp, wind) and month, wday, year
m1 <- feols(cancel_rate ~ smoke_pm25 + tmax + prcp + wind |
              Origin^month + wday + year,
            data = panel, cluster = ~ Origin + FlightDate)

m2 <- fepois(n_cancel ~ smoke_pm25 + tmax + prcp + wind |
               Origin^month + wday + year,
             offset = ~ log(n_sched),
             data = panel, cluster = ~ Origin + FlightDate)

etable(m1, m2)

Clustering by state leaves only nine clusters, too few for reliable standard errors. fixest runs the same within estimator as plm in Lecture 7, but handles many fixed effects and two-way clustering.

Weather controls and a total-effect question

Canadian fires Smoke PM2.5 Visibility Cancellations > 1,000 km away: not caused locally Weather pattern Confounder → control for it Mediator → do not control demand and labor channels

Candidate controls: precipitation, temperature, and wind. Explain their proposed roles in the causal model.

For a total-effect estimand: avoid conditioning on a variable that mediates smoke’s effect. Justify the role of visibility and distinguish it from other weather controls.

Event study on the course bundle’s episode list

smoke_episodes_2006_2025.csv already lists every regional episode with its dates and a canadian_dominated flag. Days −7 to −2 illustrate a lead test; days 0 to 7 describe the post-episode response.

episodes <- read_csv("wildfire_data/clean/smoke_episodes_2006_2025.csv") |>
  filter(region_type == "Census region", region == "Northeast",
         canadian_dominated)

windows <- episodes |>
  reframe(FlightDate = start_date + (-7:14), rel_day = -7:14,
          .by = episode_id)

es_panel <- panel |>
  inner_join(windows, by = "FlightDate", relationship = "many-to-many")

es <- feols(cancel_rate ~ i(rel_day, ref = -1) | Origin^episode_id + wday,
            data = es_panel, cluster = ~ Origin + FlightDate)

iplot(es)

Two episodes less than three weeks apart produce overlapping windows: trim them, or keep each day with its nearest episode, and say which.

Prediction and effect heterogeneity

Effect heterogeneity

  • Define which smoke-effect comparison is of interest.
  • Pre-specify interactions, such as population-density groups or hub size.
  • Assess whether the available smoke episodes support those comparisons.

Prediction

  • Ask whether smoke features improve predictions beyond weather and calendar variables.
  • Compare models on held-out years using the same observations and outcome.
  • A model that predicts cancellations accurately need not identify smoke’s causal effect.

Validation checks and their limits

Check Purpose Limitation to examine
Leads before an episode Look for changes preceding measured exposure A one-day lead may still lie inside a persistent smoke event
A justified placebo outcome Probe a mechanism or design assumption Explain why smoke should not affect the selected outcome
Hold out a state Check geographic sensitivity Exposure and flights may be concentrated in a few states
Hold out a year Check temporal sensitivity Extreme seasons may differ from the training years

Pre-specified checks

State what result would challenge the interpretation before inspecting the estimate.

🗓️ Implementation

Design risks

Exposure and outcomes

  • Few effective treatment events (2023, 2025)
  • Daily smoke vs. quarterly QCEW
  • Airports, agencies, and ports do not “belong” to one county
  • Network spillovers across airports

Coverage and sample

  • The COVID window and the post-2020 shift in transit demand
  • Changes over time in which carriers report to BTS
  • Data volume: 20 years of flight-level files
  • Overlapping episode windows in the event study

An illustrative implementation sequence

Phase 1

BTS only. Download, filter to Northeast airports, build the airport-day panel, merge smoke and the Canadian share. Figures A and B; Map 1.

Phase 2

Baseline. Two-way-clustered regressions; the event study on the episode list; lead and placebo-outcome tests.

Phase 3

QCEW. NAICS 481, 484, 485 by county-quarter, with BMZ (2024) timing. Robustness with HMS and Childs smoke.

Phase 4

Extend and write. Heterogeneity interactions, the ML prediction comparison, a first draft of results and limitations.

Complete and check one primary outcome before adding a new mode, frequency, or data source. The phases are examples, not deadlines.

Implementation checklist

  1. Build and audit one primary analysis panel.
  2. Produce an episode figure, a binned comparison, and a documented exposure map.
  3. Match the research claim to the observed outcome and exposure.
  4. Document the sample period and sensitivity checks.
  5. Justify the fixed effects, adjustment set, and dependence assumptions.
  6. Evaluate prediction separately from causal interpretation.

Design questions

Scope

Which single outcome can the available data measure and the proposed design support?

Mechanism

What additional evidence could distinguish a visibility mechanism from changes in demand?

Feasibility

Which input, definition, or join is the main feasibility risk for the first panel?

References

Borgschulte, Mark, David Molitor, and Eric Yongchen Zou. 2024. “Air Pollution and the Labor Market: Evidence from Wildfire Smoke.” Review of Economics and Statistics 106 (6): 1558–75. https://doi.org/10.1162/rest_a_01243.
Burke, Marshall, Sam Heft-Neal, Jessica Li, et al. 2022. “Exposures and Behavioural Responses to Wildfire Smoke.” Nature Human Behaviour 6 (10): 1351–61. https://doi.org/10.1038/s41562-022-01396-6.
Heutel, Garth, Nolan H. Miller, and David Molitor. 2021. “Adaptation and the Mortality Effects of Temperature Across U.S. Climate Regions.” Review of Economics and Statistics 103 (4): 740–53. https://doi.org/10.1162/rest_a_00936.
Neidell, Matthew. 2009. “Information, Avoidance Behavior, and Health: The Effect of Ozone on Asthma Hospitalizations.” Journal of Human Resources 44 (2): 450–78. https://doi.org/10.1353/jhr.2009.0018.
Schlenker, Wolfram, and W. Reed Walker. 2016. “Airports, Air Pollution, and Contemporaneous Health.” Review of Economic Studies 83 (2): 768–809. https://doi.org/10.1093/restud/rdv043.