Shared research project · Data guide

Wildfire smoke data: from raw files to clean panels

Every dataset for the Canadian wildfire smoke project, in two forms: the raw files as the agencies publish them, and clean county tables built from them with the tidyverse. Each section explains what the data measure, shows the cleaning code step by step, and describes every variable.

Important

No health outcomes. The bundle contains fires, smoke, air quality, weather, population, and boundaries only. It contains no mortality data, so course work stays outcome-blind, as the research guide requires.

Get the data

All files live in one shared Google Drive folder, wildfire_data:

Open the wildfire_data folder on Google Drive ↗

1. Open the folder and download it. Google Drive splits large folders into several zip files; unzip all of them into the same place.
2. Put the wildfire_data folder in the root of your RStudio project, next to your .Rproj file, so these paths work:

your-project/
├── your-project.Rproj
└── wildfire_data/
    ├── raw/     ← files as published by NASA, NOAA, EPA, Census, ...
    └── clean/   ← tidy county tables built by the code in this guide

3. You rarely need everything. Most projects need only clean/ (1.6 GB). Download raw/ (3.7 GB) only for the sources whose cleaning you want to reproduce or change.

If you prefer to download from R, the googledrive package can fetch single files from a link-shared folder without signing in:

library(googledrive)
drive_deauth()  # the folder is shared by link, so no Google sign-in is needed

folder <- as_id("https://drive.google.com/drive/folders/1ahnjyrwAU_YePkB99XzDeNyI653TRe_7?usp=share_link")
files  <- drive_ls(folder, recursive = TRUE)

# Example: download the monthly county panel into wildfire_data/clean/
dir.create("wildfire_data/clean", recursive = TRUE, showWarnings = FALSE)
files |>
  filter(name == "wildfire_county_month_panel_2006_2025.csv.gz") |>
  drive_download(path = "wildfire_data/clean/wildfire_county_month_panel_2006_2025.csv.gz",
                 overwrite = TRUE)

Set up R

Every code chunk on this page assumes the packages and paths below. The code chunks are shown but not run on this page; run them in your own project. Run a section’s chunks in order, because later chunks use objects made by earlier ones. Sections that build on other sections say so; most use only the files that the first section writes to clean/.

library(tidyverse)
library(sf)       # spatial joins: boundaries, FIRMS, HMS, AQS, CSU, EMBER, NBAC
library(ncdf4)    # NetCDF grids: CSU smoke sample and EMBER model runs
library(readxl)   # OMB metro-area delineation (.xlsx)
library(foreign)  # NBAC attribute table (.dbf)

raw_dir   <- "wildfire_data/raw"
clean_dir <- "wildfire_data/clean"

Conventions used in every clean file

• Counties. GEOID is the 5-character county FIPS code, stored as text ("09001", not 9001). Always read it with col_types = cols(GEOID = col_character()) or the leading zero disappears. The county universe is the 3,108 counties of the contiguous United States plus DC, on 2020 Census boundaries, so Connecticut keeps its 8 legacy counties.
• Dates. date is a calendar day in ISO form (2023-06-07). Satellite products use UTC (Coordinated Universal Time) days; fire detections also carry a local solar day; monitor data use the local calendar day. Each section says which one it uses.
• Units. PM2.5 in micrograms per cubic meter (µg/m³), temperature in °C, precipitation in mm, fire radiative power (FRP) in megawatts (MW), area in km² or hectares.
• Missing vs. zero. A missing row or NA can mean “zero”, “not observed”, or “not applicable”. Each section states which, and the panel section shows how the joins handle it.
• Big daily tables are split into one .csv.gz file per year inside a folder. read_csv() reads .gz files directly.

Data map

Source	What it measures	Raw grain	Clean grain	Years
Boundaries and population	County, state, and province outlines; 2020 population; metro areas	polygon; block group	county	2020 / 2021 / 2023
SEER population	County population by age and sex	county × year × race × sex × age	county × year	2006–2024
NASA FIRMS	Satellite fire detections and fire radiative power	detection	detection; province × day; country × day	2003–2024
CWFIS NBAC	Canadian burned area by fire	fire polygon	fire; province × year	1972–2025
NOAA HMS	Analyst-drawn smoke plume outlines	plume polygon	plume; county × day; county × month	2005–2025
CSU kriged smoke	Smoke PM2.5 near the ground (main measure)	15 km grid cell × day	county × day; county × month	2006–2025
Childs et al.	Smoke PM2.5 (machine-learning estimate)	county × smoke day	county × day; county × month	2006–2023
EPA EMBER	Modeled fire and Canadian-fire PM2.5	36 km grid cell × day	county × day	Apr–Sep 2023
EPA AQS	Monitor PM2.5 and ozone	monitor × day	site × day; county × day; county × month	2006–2025
nClimGrid-Daily	Temperature and precipitation	county × month file, one column per day	county × day; county × month	2006–2025
ERA5	Winds, humidity, boundary-layer height	0.25° grid (not shipped)	county × day; county × month	2006–2025
Smoke episodes	Multi-day smoke episodes; Canadian share of nearby fires	project-built	episode; county × day	2006–2025
County panels	Everything joined	—	county × month; county × day (2023)	2006–2025

What is in the folder

Raw folder	Files	Size
raw/aqs/	41	188 MB
raw/boundaries/	4	149 MB
raw/census_bg/	1	10 MB
raw/childs/	2	115 MB
raw/csu/	2	491 MB
raw/ember/	5	100 MB
raw/firms/	67	445 MB
raw/hms/	21	85 MB
raw/nbac/	3	1.2 GB
raw/nclimgrid/	724	751 MB
raw/seer/	2	271 MB
All raw files		3.7 GB

Clean file or folder	Files	Size
clean/aqs_county_day_2006_2025.csv.gz	1	25 MB
clean/aqs_county_month_2006_2025.csv	1	6 MB
clean/aqs_ozone_site_day_2006_2025.csv.gz	1	27 MB
clean/aqs_pm25_site_day_2006_2025.csv.gz	1	18 MB
clean/aqs_sites.csv	1	237 KB
clean/canada_provinces_2021.csv	1	1 KB
clean/canada_provinces_2021.gpkg	1	1 MB
clean/canadian_dominated_summary.csv	1	15 KB
clean/census_bg_centers_2020.csv	1	10 MB
clean/childs_smoke_county_day/	18 files (one per year)	58 MB
clean/childs_smoke_county_month_2006_2023.csv	1	19 MB
clean/counties_2020.csv	1	334 KB
clean/counties_2020.gpkg	1	18 MB
clean/county_cbsa_2023.csv	1	256 KB
clean/county_day_fire_share_2006_2025.csv.gz	1	55 MB
clean/csu_sample_county_day_2023.csv.gz	1	7 MB
clean/csu_smoke_county_day/	20 files (one per year)	119 MB
clean/csu_smoke_county_month_2006_2025.csv	1	39 MB
clean/ember_county_day_2023.csv	1	26 MB
clean/era5_county_day/	20 files (one per year)	545 MB
clean/era5_county_month_2006_2025.csv	1	51 MB
clean/firms_canada_province_day_2003_2024.csv	1	3 MB
clean/firms_country_day_2003_2024.csv	1	901 KB
clean/firms_modis_detections_2003_2024.csv.gz	1	107 MB
clean/hms_calendar_2005_2025.csv	1	158 KB
clean/hms_county_day_2006_2025.csv.gz	1	15 MB
clean/hms_county_month_2006_2025.csv	1	21 MB
clean/hms_smoke_plumes_2005_2025.csv	1	50 MB
clean/nbac_fires_1972_2025.csv	1	7 MB
clean/nbac_province_year_1972_2025.csv	1	32 KB
clean/nclimgrid_county_day/	20 files (one per year)	239 MB
clean/nclimgrid_county_month_2006_2025.csv	1	42 MB
clean/seer_county_year_2006_2024.csv	1	4 MB
clean/smoke_episodes_2006_2025.csv	1	41 KB
clean/smoke_person_days_by_bin.csv	1	79 KB
clean/us_states_2020.csv	1	4 KB
clean/us_states_2020.gpkg	1	416 KB
clean/wildfire_county_day_2023.csv.gz	1	38 MB
clean/wildfire_county_month_panel_2006_2025.csv.gz	1	51 MB
All clean files		1.6 GB

Boundaries, population centers, and metro areas

These files say where places are and how many people live in them. The U.S. Census Bureau publishes 2020 county outlines in its cartographic boundary files, which are simplified for maps, and 2020 state outlines in TIGER/Line, its full-detail geographic database. For each of about 240,000 block groups (small areas of roughly 600 to 3,000 people), it also publishes the 2020 Census population and its center of population. Statistics Canada publishes the outlines of the 13 provinces and territories from its 2021 Census, and the Office of Management and Budget (OMB) groups U.S. counties into metropolitan and micropolitan areas.

Every other module builds on these files. Each county panel has one row per county in counties_2020.csv, the block-group file supplies population weights, and the state and province outlines label fire detections. They are snapshots of 2020–2023: they do not show how population or boundaries changed later, and they say nothing about where smoke goes.

Provider	U.S. Census Bureau (county and state boundaries, centers of population, delineation file); Statistics Canada (provinces and territories); OMB (metro and micro area definitions)
Coverage	2020 Census (U.S.), 2021 Census (Canada), July 2023 metro delineation · raw: all U.S. states, DC, Puerto Rico, the Island Areas (American Samoa, Guam, Northern Mariana Islands, U.S. Virgin Islands), and Canada; clean: contiguous U.S. + DC (counties, block groups), 50 states + DC, Canada
Raw unit	one polygon = a county, a state, or a province; one row = a block group (centers file) or a county in a metro or micro area (delineation file)
Clean unit	one row = a county (3,108), a block group (238,193), a state (51), a province or territory (13), or a county with its metro area (3,108)
Licence and citation	U.S. files: public domain (U.S. Government work); cite “U.S. Census Bureau, 2020 Cartographic Boundary Files, 2020 TIGER/Line Shapefiles, and 2020 Centers of Population” and “Office of Management and Budget, Bulletin No. 23-01 (July 2023)”. Canadian file: Statistics Canada Open Licence; cite “Statistics Canada, 2021 Census Boundary Files, Provinces and territories (Cartographic), catalogue 92-160-X”
Documentation	Census cartographic boundary files; TIGER/Line shapefiles; 2020 centers of population and method; StatCan boundary files reference guide; Census delineation files and glossary; Connecticut: OPM municipalities by planning region and Census notice on Connecticut’s county equivalents

Raw files

File (in wildfire_data/raw/)	Size	Format	One row is
boundaries/cb_2020_us_county_500k.zip	12.0 MB	zipped ESRI shapefile, NAD83 longitude/latitude (EPSG:4269)	one county or county equivalent (3,234 features: all states, DC, Puerto Rico, Island Areas)
boundaries/tl_2020_us_state.zip	9.3 MB	zipped ESRI shapefile, NAD83 longitude/latitude (EPSG:4269)	one state, DC, or territory (56 features)
boundaries/lpr_000b21a_e.zip	127.5 MB	zipped ESRI shapefile, Statistics Canada Lambert in meters (EPSG:3347)	one province or territory (13 features)
census_bg/CenPop2020_Mean_BG__dl20260927.txt	10.1 MB	comma-separated text, UTF-8 with a byte-order mark	one block group (242,335 rows: 50 states, DC, Puerto Rico)
boundaries/list1_2023.xlsx	0.1 MB	Excel workbook, sheet “List 1”, every cell stored as text	one county or county equivalent inside a metro or micro area (1,915 rows)

Raw variables

County shapefile cb_2020_us_county_500k.zip (Census 2020 cartographic boundary file, 1:500,000), described by the Census metadata file cb_2020_us_county_500k.shp.ea.iso.xml inside the zip.

Variable	Type	Description
STATEFP	Character (2)	State FIPS code (Federal Information Processing Series: the federal number of the state), e.g., 36 = New York
COUNTYFP	Character (3)	County FIPS code within the state, e.g., 055
COUNTYNS	Character (8)	County code in the Geographic Names Information System (GNIS) of the U.S. Geological Survey
AFFGEOID	Character (14)	American FactFinder identifier: summary level code + geovariant code + 00US + GEOID, e.g., 0500000US36055
GEOID	Character (5)	County identifier: state FIPS + county FIPS, e.g., 36055 = Monroe County, New York. Keep it as text so the leading zero of codes such as 01001 survives
NAME	Character (100)	County name, e.g., Monroe
NAMELSAD	Character (100)	County name with its legal/statistical description, e.g., Monroe County, Orleans Parish, Baltimore city
STUSPS	Character (2)	U.S. Postal Service state abbreviation
STATE_NAME	Character (100)	State name
LSAD	Character (2)	Legal/statistical area description code: 00 blank, 03 City and Borough, 04 Borough, 05 Census Area, 06 County, 07 District, 10 Island, 12 Municipality, 13 Municipio, 15 Parish, 25 city
ALAND	Numeric (14)	Land area, square meters
AWATER	Numeric (14)	Water area inside the county’s legal boundary, square meters (includes Great Lakes and coastal water)
geometry	MULTIPOLYGON	County outline, generalized for 1:500,000 maps and clipped to the shoreline

State shapefile tl_2020_us_state.zip (TIGER/Line 2020), described by the Census metadata file tl_2020_us_state.shp.ea.iso.xml inside the zip.

Variable	Type	Description
REGION	Character (2)	Census region code: 1 Northeast, 2 Midwest, 3 South, 4 West, 9 Puerto Rico and the Island Areas
DIVISION	Character (2)	Census division code: 0 Puerto Rico and the Island Areas, 1 New England, 2 Middle Atlantic, 3 East North Central, 4 West North Central, 5 South Atlantic, 6 East South Central, 7 West South Central, 8 Mountain, 9 Pacific
STATEFP	Character (2)	State FIPS code
STATENS	Character (8)	State GNIS code
GEOID	Character (2)	State identifier (same as STATEFP)
STUSPS	Character (2)	U.S. Postal Service state abbreviation
NAME	Character (100)	State name
LSAD	Character (2)	Legal/statistical area description code; 00 (blank) for every state
MTFCC	Character (5)	Feature class code from MAF/TIGER, the Census Bureau’s address list (Master Address File) joined to its geographic database; G4000 = state or equivalent feature
FUNCSTAT	Character (1)	Functional status; A = active government providing primary general-purpose functions
ALAND	Numeric (14)	Land area, square meters
AWATER	Numeric (14)	Water area, square meters
INTPTLAT	Character (11)	Latitude of the Census internal point (a point inside the state), stored as text with a sign, e.g., +38.6472854
INTPTLON	Character (12)	Longitude of the internal point, e.g., -080.6183274
geometry	MULTIPOLYGON	Full-detail state outline; it includes inland water, coastal water, and the state’s part of the Great Lakes

Province shapefile lpr_000b21a_e.zip (Statistics Canada 2021 cartographic boundary file), described by Table 4.1 of the StatCan reference guide.

Variable	Type	Description
PRUID	Character (2)	Uniquely identifies a province or territory, e.g., 35 = Ontario
DGUID	Character (21)	Dissemination Geography Unique Identifier, e.g., 2021A000235 (2021 vintage, province schema, PRUID 35)
PRNAME	Character (100)	Province or territory name, in English and French when they differ, e.g., Quebec / Québec
PRENAME	Character (100)	Name in English
PRFNAME	Character (100)	Name in French
PREABBR	Character (10)	English abbreviation, e.g., Que.
PRFABBR	Character (10)	French abbreviation, e.g., Qc
LANDAREA	Numeric (12, 4)	Land area, square kilometers
geometry	MULTIPOLYGON	Outline with coastal water erased, so coasts become thousands of separate pieces. Inland lakes such as Lake Winnipeg stay inside; the Great Lakes and the St. Lawrence are erased as coastal water

Block-group centers file CenPop2020_Mean_BG__dl20260927.txt, the Census file CenPop2020_Mean_BG.txt with the research project’s download date (2026-09-27) added to its name.

Variable	Type	Description
STATEFP	text, 2 characters	State FIPS code
COUNTYFP	text, 3 characters	County FIPS code
TRACTCE	text, 6 characters	Census tract code
BLKGRPCE	text, 1 character	Block group code within the tract
POPULATION	integer	2020 Census population of the block group (April 1, 2020); 0 for 1,393 block groups, which still get a coordinate (how it is chosen is not documented by the provider)
LATITUDE	decimal degrees, signed text, e.g., +32.464466	Latitude of the block group’s center of population: the point where the area would balance if every resident had equal weight. Census computes it from a point inside each census block, weighted by block population
LONGITUDE	decimal degrees, e.g., -086.486302	Longitude of the center of population (negative = west)

Delineation file list1_2023.xlsx, sheet “List 1” (rows 1 and 2 are titles, row 3 holds the column names, and the last three rows are a blank line, a note, and a source line), with definitions from the Census glossary.

Variable	Type	Description
CBSA Code	text (5 digits)	Core Based Statistical Area (CBSA) code. A CBSA is a metropolitan or micropolitan statistical area
Metropolitan Division Code	text (5 digits), blank if none	Code of the metropolitan division: a county or group of counties inside a metro area whose core has at least 2.5 million people
CSA Code	text (3 digits), blank if none	Combined Statistical Area code: two or more adjacent CBSAs linked by commuting
CBSA Title	text	CBSA name, e.g., Rochester, NY
Metropolitan/Micropolitan Statistical Area	text	Metropolitan Statistical Area (core urban area of at least 50,000 people) or Micropolitan Statistical Area (core urban area of 10,000 to 49,999)
Metropolitan Division Title	text	Name of the metropolitan division
CSA Title	text	Name of the CSA
County/County Equivalent	text	County name; for Connecticut, a planning region name
State Name	text	State name
FIPS State Code	text (2 digits)	State FIPS code
FIPS County Code	text (3 digits)	County FIPS code; for Connecticut, a planning region code 110–190
Central/Outlying County	text	Central: the county holds all or much of the core urban area. Outlying: at least a quarter of its workers commute to the central counties, or at least a quarter of its jobs are held by commuters from them

Cleaning steps

The script R/m00_places.R makes these decisions:

1. County frame. Keep the 3,108 counties of the contiguous U.S. plus DC from the 2020 cartographic file. Its outlines stop at the shoreline, so a county polygon does not reach into Lake Michigan or the ocean. These are 2020 boundaries, so Connecticut keeps its 8 old counties.
2. Groups typed in, then checked. Census regions, Census divisions, and the project’s eastern split are typed in as small tables with tribble() so you can see them. The script stops if a typed division disagrees with the code in the Census state file.
3. Population from block groups. County and state populations are sums of block-group populations. A county’s representative point (lat, lon) is its population-weighted center, computed with the Census formula from block-group centers. It is not a geometric center made with st_point_on_surface().
4. One CRS. A coordinate reference system (CRS) says how coordinates map to places on Earth; EPSG codes number them. Every .gpkg file is converted to longitude/latitude on the WGS 84 datum (EPSG:4326), the CRS of the fire and smoke data. The Census files use NAD83 (EPSG:4269), which differs from WGS 84 by about one meter in the U.S.
5. Light, gap-free outlines. State outlines are simplified with a 500 m tolerance and province outlines with 1 km. “Tolerance” is the largest distance a simplified border may move from the original. A shared border is simplified once for both neighbors, so no gaps or overlaps open between two states or between two provinces. Canadian islands smaller than 1 km² are dropped first, and water enclosed by a single province (91 small holes, about 370 km² in total) becomes part of that province. The U.S. and Canadian layers come from different agencies and are simplified separately, so they do not share a border (see “Checks and cautions”).
6. Province file read in pieces. The raw Canadian file is so detailed that reading it whole takes about 1.7 GB of memory. The script reads and shrinks one province at a time, which needs about 1 GB.
7. Metro areas from the 2023 delineation. Each county gets its CBSA, metropolitan division, and CSA. Connecticut counties take the CBSA of their dominant planning region (chunks m00-ct-regions to m00-ct-notes).

The first chunk creates the output folder and lists the state codes outside the contiguous U.S. Puerto Rico and the four Island Areas are territories, not states, so territories lists them on their own; the state file drops them.

# Every clean file of this module goes straight into wildfire_data/clean/.
dir.create(clean_dir, recursive = TRUE, showWarnings = FALSE)

# State FIPS codes outside the contiguous U.S.: Alaska, Hawaii, American
# Samoa, Guam, Northern Mariana Islands, Puerto Rico, U.S. Virgin Islands.
non_contiguous <- c("02", "15", "60", "66", "69", "72", "78")
# Puerto Rico and the four Island Areas: territories, not states.
territories <- c("60", "66", "69", "72", "78")

The next chunk types in the Census regions and divisions and the project’s eastern split, then turns the space-separated state lists into one row per state.

# Census regions and divisions, typed in from the Census Bureau's official
# list so the grouping is visible. The next chunk checks every state against
# the division code stored in the Census state file.
census_divisions <- tribble(
  ~division_code, ~census_division,     ~census_region, ~states,
  "1",            "New England",        "Northeast",    "CT ME MA NH RI VT",
  "2",            "Middle Atlantic",    "Northeast",    "NJ NY PA",
  "3",            "East North Central", "Midwest",      "IL IN MI OH WI",
  "4",            "West North Central", "Midwest",      "IA KS MN MO NE ND SD",
  "5",            "South Atlantic",     "South",        "DE DC FL GA MD NC SC VA WV",
  "6",            "East South Central", "South",        "AL KY MS TN",
  "7",            "West South Central", "South",        "AR LA OK TX",
  "8",            "Mountain",           "West",         "AZ CO ID MT NV NM UT WY",
  "9",            "Pacific",            "West",         "AK CA HI OR WA"
)

# The project's eastern split: four groups of eastern states used in its
# descriptive tables of Canadian smoke. It is NOT a Census grouping: its
# "Mid-Atlantic" adds DE, DC, and MD to the Census Middle Atlantic division.
east_split_states <- tribble(
  ~east_split,     ~states,
  "New England",   "CT ME MA NH RI VT",
  "Mid-Atlantic",  "DE DC MD NJ NY PA",
  "Great Lakes",   "IL IN MI OH WI",
  "Upper Midwest", "IA MN ND SD"
)

# One row per state: split the space-separated lists into rows, then join.
state_groups <- census_divisions |>
  separate_longer_delim(states, delim = " ") |>
  rename(state_abbr = states) |>
  left_join(
    east_split_states |>
      separate_longer_delim(states, delim = " ") |>
      rename(state_abbr = states),
    by = "state_abbr"
  ) |>
  mutate(east_split = replace_na(east_split, "Other"))

stopifnot(nrow(state_groups) == 51, !anyDuplicated(state_groups$state_abbr))

Read the state file straight from its zip and check the typed-in divisions against the Census codes. GDAL is the library that read_sf() uses to open spatial files.

# TIGER/Line 2020 states. TIGER/Line is the Census Bureau's full-detail
# geographic database. GDAL reads a shapefile inside a .zip directly when
# the path starts with /vsizip/, so nothing needs to be unzipped.
states_raw <- read_sf(file.path(
  "/vsizip", raw_dir, "boundaries", "tl_2020_us_state.zip", "tl_2020_us_state.shp"
))

# Keep the 50 states and DC, and attach the groups typed in above.
states <- states_raw |>
  filter(!STATEFP %in% territories) |>
  select(STATEFP, state_abbr = STUSPS, state_name = NAME, DIVISION, ALAND, AWATER) |>
  left_join(state_groups, by = "state_abbr")

# The typed-in division of every state must equal the Census code.
stopifnot(
  nrow(states) == 51,
  !anyNA(states$census_division),
  all(states$DIVISION == states$division_code)
)

Read the county outlines and keep the contiguous counties as counties_frame. Land and water areas are converted from square meters to km².

# Census 2020 cartographic boundary file (1:500,000). Its county outlines are
# clipped to the shoreline, so the Great Lakes and coastal waters are not
# inside the county polygons (TIGER/Line county files extend into them).
counties_raw <- read_sf(file.path(
  "/vsizip", raw_dir, "boundaries", "cb_2020_us_county_500k.zip",
  "cb_2020_us_county_500k.shp"
))

counties_frame <- counties_raw |>
  filter(!STATEFP %in% non_contiguous) |>
  transmute(
    GEOID, county_name = NAMELSAD, STATEFP, COUNTYFP,
    state_abbr = STUSPS, state_name = STATE_NAME,
    # ALAND and AWATER are in square meters; 1 km2 = 1,000,000 m2.
    land_km2 = round(ALAND / 1e6, 2),
    water_km2 = round(AWATER / 1e6, 2)
  )

# 3,108 counties and county equivalents: 48 states plus DC. Connecticut keeps
# its 8 legacy counties because these are 2020 boundaries.
stopifnot(nrow(counties_frame) == 3108, !anyDuplicated(counties_frame$GEOID))

Read the block-group centers. Declaring every FIPS column as character keeps codes such as 01 and 001 intact.

# Census 2020 centers of population, one row per block group (a small area
# of about 600 to 3,000 people). The file is a plain CSV despite its .txt
# name. FIPS codes must be read as text to keep their leading zeros.
bg_raw <- read_csv(
  file.path(raw_dir, "census_bg", "CenPop2020_Mean_BG__dl20260927.txt"),
  col_types = cols(
    STATEFP = col_character(), COUNTYFP = col_character(),
    TRACTCE = col_character(), BLKGRPCE = col_character(),
    POPULATION = col_integer(), LATITUDE = col_double(), LONGITUDE = col_double()
  )
)

bg_all <- bg_raw |>
  transmute(
    # 2 (state) + 3 (county) + 6 (tract) + 1 (block group) = 12 characters
    bg_geoid = str_c(STATEFP, COUNTYFP, TRACTCE, BLKGRPCE),
    GEOID = str_c(STATEFP, COUNTYFP),
    STATEFP,
    pop_2020 = POPULATION,
    lat = LATITUDE,
    lon = LONGITUDE
  )

stopifnot(all(str_length(bg_all$bg_geoid) == 12), !anyDuplicated(bg_all$bg_geoid))

Keep the contiguous block groups and write them. The checks confirm the row count, that every block group belongs to a frame county, that every county has block groups, and that every center lies inside the contiguous U.S. box of latitudes and longitudes.

# Keep the block groups of the 3,108 contiguous counties. Every one of them
# must belong to a county in the frame, and every county must have some.
bg_centers <- bg_all |>
  filter(!STATEFP %in% non_contiguous) |>
  select(bg_geoid, GEOID, pop_2020, lat, lon) |>
  arrange(bg_geoid)

stopifnot(
  nrow(bg_centers) == 238193,
  all(bg_centers$GEOID %in% counties_frame$GEOID),
  all(counties_frame$GEOID %in% bg_centers$GEOID),
  all(bg_centers$pop_2020 >= 0),
  all(between(bg_centers$lat, 24, 50)), all(between(bg_centers$lon, -125, -66))
)

# Grain: one row per block group (bg_geoid is unique).
write_csv(bg_centers, file.path(clean_dir, "census_bg_centers_2020.csv"), na = "")

Sum population by county and compute each county’s population-weighted center. The longitude formula weights by the cosine of latitude, as the Census Bureau does. The weight is computed first, in its own column, and the county results get new names, so the order of the lines inside summarize() cannot change the answer.

# County population is the sum over its block groups. The county's
# population-weighted center is the average of block-group centers, each
# weighted by its population: it sits where people live, not at the middle
# of the county's land. We use the Census Bureau's formula: longitude is also
# weighted by cos(latitude), because a degree of longitude gets shorter
# toward the North Pole.
county_pop <- bg_centers |>
  # Longitude weight of each block group: population x cos(latitude).
  mutate(w_lon = pop_2020 * cos(lat * pi / 180)) |>
  group_by(GEOID) |>
  # New names for the county results, so no line reads a county value
  # where it needs a block-group value.
  summarize(
    center_lat = weighted.mean(lat, w = pop_2020),
    center_lon = weighted.mean(lon, w = w_lon),
    total_pop = sum(pop_2020),
    .groups = "drop"
  ) |>
  transmute(GEOID, lat = round(center_lat, 5), lon = round(center_lon, 5),
            pop_2020 = total_pop)

# Every county has residents, so every weighted mean is defined.
stopifnot(nrow(county_pop) == 3108, all(county_pop$pop_2020 > 0))

Assemble the county table as a new object, counties_sf, so that counties_frame stays unchanged and the chunk can be run again; then convert it to EPSG:4326, check it, and write the CSV and GeoPackage files. A GeoPackage (.gpkg) is one file that holds both the table and the outlines; the chunk reads it back with read_sf() and counts its rows, because a GeoPackage write can fail partway on some drives (see “Checks and cautions”).

# The result gets a new name, so its input (counties_frame) stays unchanged
# and this chunk gives the same answer when you run it again.
counties_sf <- counties_frame |>
  left_join(st_drop_geometry(states) |>
              select(state_abbr, census_region, census_division, east_split),
            by = "state_abbr") |>
  left_join(county_pop, by = "GEOID") |>
  select(GEOID, county_name, STATEFP, COUNTYFP, state_abbr, state_name,
         census_region, census_division, east_split,
         land_km2, water_km2, lat, lon, pop_2020) |>
  arrange(GEOID) |>
  # NAD83 (the Census datum) to WGS 84 (EPSG:4326), the CRS of the fire and
  # smoke data; the two differ by about a meter in the U.S.
  st_transform(4326)

# Grain: one row per county; no missing values in any column.
# 329,260,619 is the official 2020 Census count for the 48 states and DC.
stopifnot(
  nrow(counties_sf) == 3108, !anyDuplicated(counties_sf$GEOID),
  all(str_length(counties_sf$GEOID) == 5),
  !anyNA(st_drop_geometry(counties_sf)),
  all(between(counties_sf$lat, 24, 50)), all(between(counties_sf$lon, -125, -66)),
  all(counties_sf$land_km2 > 0), all(counties_sf$water_km2 >= 0),
  sum(counties_sf$pop_2020) == 329260619,
  all(st_is_valid(counties_sf))
)

write_csv(st_drop_geometry(counties_sf), file.path(clean_dir, "counties_2020.csv"),
          na = "")
write_sf(counties_sf, file.path(clean_dir, "counties_2020.gpkg"), delete_dsn = TRUE)
# A GeoPackage is a small database, and on some drives a write can fail
# partway, so read the file back and count its rows.
stopifnot(nrow(read_sf(file.path(clean_dir, "counties_2020.gpkg"))) == 3108)

count(st_drop_geometry(counties_sf), east_split)

The next two chunks define helper functions. The first simplifies polygons that share borders without opening gaps or overlaps; you do not need every line of it to use the outputs.

# Simplify a set of polygons that share borders (states, provinces) without
# opening gaps or overlaps between neighbors. Simplifying each polygon on its
# own would move a shared border differently on its two sides, so a fire
# detection near the border could land in two polygons or in none. Instead:
#  1. take all outlines as lines and merge them, so each shared border is
#     stored once;
#  2. simplify each line (tolerance in the units of the CRS, here meters);
#  3. rebuild polygons ("faces") from the simplified lines;
#  4. give each face to the unit that contains a point inside it, or, when no
#     unit does (a thin island or a sliver), to the nearest unit.
# Side effect of step 4: a hole (water enclosed by one unit) has no owner,
# so it goes to the nearest unit and becomes part of the surrounding polygon.
# x must use a projected CRS (meters), not longitude/latitude.
simplify_shared_borders <- function(x, id, tolerance) {
  lines <- x |>
    st_geometry() |>
    st_boundary() |>
    st_union() |>
    st_line_merge() |>
    st_cast("LINESTRING") |>
    st_simplify(preserveTopology = TRUE, dTolerance = tolerance)
  faces <- st_union(lines) |>
    st_polygonize() |>
    st_collection_extract("POLYGON")
  faces <- st_sf(geometry = faces)
  inside <- st_intersects(st_point_on_surface(faces), x)
  owner <- map_int(inside, \(i) if (length(i) > 0) i[1] else NA_integer_)
  orphan <- is.na(owner)
  owner[orphan] <- st_nearest_feature(faces[orphan, ], x)
  faces[[id]] <- x[[id]][owner]
  faces |>
    group_by(across(all_of(id))) |>
    summarize(.groups = "drop") |>
    st_cast("MULTIPOLYGON")
}

The second converts a layer to longitude/latitude and repairs any tiny polygon that the conversion made invalid. The repair uses GEOS, the library that sf uses for flat (planar) geometry, so it needs s2 (the spherical geometry engine that sf uses for longitude/latitude data) switched off. sf_use_s2() is a global setting, so on.exit() restores it when the function ends, even after an error.

# Convert to longitude/latitude (EPSG:4326). Reprojection can make a tiny
# simplified island cross itself, so repair with planar (GEOS) rules, then
# confirm validity under the spherical (s2) rules that sf uses for lon/lat.
to_lonlat <- function(x) {
  n_rows <- nrow(x)
  # sf_use_s2() is a global switch. on.exit() puts back the old setting when
  # the function ends, even if a step fails, so s2 never stays off by mistake.
  old_s2 <- suppressMessages(sf_use_s2(FALSE))
  on.exit(suppressMessages(sf_use_s2(old_s2)), add = TRUE)
  x <- st_transform(x, 4326) |>
    st_make_valid() |>
    st_collection_extract("POLYGON", warn = FALSE) |>
    st_cast("MULTIPOLYGON")
  suppressMessages(sf_use_s2(TRUE))  # check validity with the s2 rules
  stopifnot(nrow(x) == n_rows, all(st_is_valid(x)))
  x
}

Build the state table, simplify the state outlines, and write both files.

# State population comes from all block groups, including Alaska and Hawaii.
state_pop <- bg_all |>
  group_by(STATEFP) |>
  summarize(pop_2020 = sum(pop_2020), .groups = "drop")

states_out <- states |>
  left_join(state_pop, by = "STATEFP") |>
  transmute(
    STATEFP, state_abbr, state_name, census_region, census_division, east_split,
    contiguous = !STATEFP %in% non_contiguous,
    land_km2 = round(ALAND / 1e6, 2),
    water_km2 = round(AWATER / 1e6, 2),
    pop_2020
  ) |>
  arrange(STATEFP)

# Simplify with a 500 m tolerance in CONUS Albers (EPSG:5070, meters). Alaska
# and Hawaii are far from its center, so there the tolerance is only roughly 500 m.
states_shapes <- states_out |>
  select(STATEFP) |>
  st_transform(5070) |>
  simplify_shared_borders("STATEFP", tolerance = 500) |>
  to_lonlat()

states_sf <- states_out |>
  st_drop_geometry() |>
  inner_join(states_shapes, by = "STATEFP") |>
  st_as_sf()

# Grain: one row per state or DC; 49 of them are contiguous (48 + DC).
# 331,449,281 is the official 2020 Census count for the 50 states and DC.
stopifnot(
  nrow(states_sf) == 51, !anyDuplicated(states_sf$STATEFP),
  sum(states_sf$contiguous) == 49, !anyNA(st_drop_geometry(states_sf)),
  all(states_sf$land_km2 > 0), sum(states_sf$pop_2020) == 331449281
)

write_csv(st_drop_geometry(states_sf), file.path(clean_dir, "us_states_2020.csv"),
          na = "")
write_sf(states_sf, file.path(clean_dir, "us_states_2020.gpkg"), delete_dsn = TRUE)
stopifnot(nrow(read_sf(file.path(clean_dir, "us_states_2020.gpkg"))) == 51)

Define how to read one province. An SQL query (SELECT * FROM layer WHERE PRUID = '62') makes GDAL return only the matching row of the shapefile. The function splits that province into its pieces of land, drops islands under 1 km², and thins the outlines to 50 m. It returns a list with a one-row info table and the shapes.

# Statistics Canada 2021 cartographic boundary file: one multipolygon per
# province or territory, in Statistics Canada Lambert (EPSG:3347, meters).
# It is very detailed: about 176,000 pieces of land, mostly small islands
# along the coasts and in the Great Lakes (most in Nunavut, Ontario, British
# Columbia, Quebec, and Newfoundland and Labrador). Reading it all at once
# takes about 1.7 GB of memory, so we read one province at a time with an
# SQL query and shrink it before the next one.
provinces_shp <- file.path(
  "/vsizip", raw_dir, "boundaries", "lpr_000b21a_e.zip", "lpr_000b21a_e.shp"
)

# Read one province (by its code, PRUID), split it into its separate pieces
# of land, drop islands smaller than 1 km2, and thin the outlines to 50 m.
# After thinning, a shared border can differ by up to 50 m on its two sides;
# simplify_shared_borders() absorbs that difference later.
read_province <- function(pruid) {
  query <- str_glue("SELECT * FROM lpr_000b21a_e WHERE PRUID = '{pruid}'")
  parts <- read_sf(provinces_shp, query = query) |>
    # a query names the geometry column "_ogr_geometry_"; rename it
    st_set_geometry("geometry") |>
    st_cast("POLYGON", warn = FALSE) |>
    mutate(part_km2 = as.numeric(st_area(geometry)) / 1e6)
  list(
    # one row: the names, the land area, and what the 1 km2 cut removed
    info = parts |>
      st_drop_geometry() |>
      summarize(n_islands_dropped = sum(part_km2 < 1),
                km2_dropped = sum(part_km2[part_km2 < 1]),
                .by = c(PRUID, PRENAME, PRFNAME, LANDAREA)),
    shapes = parts |>
      filter(part_km2 >= 1) |>
      select(PRUID) |>
      st_simplify(preserveTopology = TRUE, dTolerance = 50)
  )
}

Run the function for the 13 provinces and territories with map(), then stack the 13 info rows and the 13 sets of shapes. This is the slowest step of the module (about 20 seconds).

# Two-letter postal abbreviations; the file only has the longer forms
# ("N.L.", "Que.").
province_codes <- tribble(
  ~PRUID, ~prov_abbr,
  "10", "NL", "11", "PE", "12", "NS", "13", "NB", "24", "QC", "35", "ON",
  "46", "MB", "47", "SK", "48", "AB", "59", "BC", "60", "YT", "61", "NT", "62", "NU"
)

# Run read_province() for the 13 provinces and territories, then stack the
# 13 info rows and the 13 sets of shapes.
provinces_read <- map(province_codes$PRUID, read_province)
province_info <- map(provinces_read, "info") |> bind_rows()
province_parts <- map(provinces_read, "shapes") |> bind_rows()
rm(provinces_read)

stopifnot(nrow(province_info) == 13,
          setequal(province_parts$PRUID, province_codes$PRUID))

# The dropped islands are about 0.05% of Canada's land.
province_info |>
  summarize(n_islands = sum(n_islands_dropped), km2 = sum(km2_dropped))

Simplify the shared provincial borders with a 1 km tolerance, add the names and two-letter abbreviations, and write the files.

provinces_shapes <- province_parts |>
  simplify_shared_borders("PRUID", tolerance = 1000) |>
  to_lonlat()

provinces_sf <- province_info |>
  transmute(PRUID, prov_name = PRENAME, prov_name_fr = PRFNAME,
            land_km2 = round(LANDAREA, 2)) |>
  left_join(province_codes, by = "PRUID") |>
  relocate(prov_abbr, .after = PRUID) |>
  inner_join(provinces_shapes, by = "PRUID") |>
  st_as_sf() |>
  arrange(PRUID)

# Grain: one row per province or territory (13); every code and two-letter
# abbreviation is present and unique.
stopifnot(
  nrow(provinces_sf) == 13, !anyDuplicated(provinces_sf$PRUID),
  !anyDuplicated(provinces_sf$prov_abbr), !anyNA(st_drop_geometry(provinces_sf)),
  all(provinces_sf$land_km2 > 0)
)

write_csv(st_drop_geometry(provinces_sf),
          file.path(clean_dir, "canada_provinces_2021.csv"), na = "")
write_sf(provinces_sf, file.path(clean_dir, "canada_provinces_2021.gpkg"),
         delete_dsn = TRUE)
stopifnot(nrow(read_sf(file.path(clean_dir, "canada_provinces_2021.gpkg"))) == 13)

Read the delineation sheet as text and drop its footer rows.

# OMB July 2023 delineation of metropolitan and micropolitan areas. Rows 1
# and 2 of the sheet are titles, so the column names are in row 3 (skip = 2).
# Reading every column as text keeps codes such as "001" intact.
cbsa_raw <- read_excel(
  file.path(raw_dir, "boundaries", "list1_2023.xlsx"),
  sheet = "List 1", skip = 2, col_types = "text"
)

delineation <- cbsa_raw |>
  rename(
    cbsa_code = `CBSA Code`, metdiv_code = `Metropolitan Division Code`,
    csa_code = `CSA Code`, cbsa_title = `CBSA Title`,
    cbsa_kind = `Metropolitan/Micropolitan Statistical Area`,
    metdiv_title = `Metropolitan Division Title`, csa_title = `CSA Title`,
    delin_name = `County/County Equivalent`, st = `FIPS State Code`,
    co = `FIPS County Code`, cbsa_central = `Central/Outlying County`
  ) |>
  # The last three rows are a blank line and two notes, with no FIPS codes.
  filter(!is.na(st), !is.na(co)) |>
  mutate(
    delin_geoid = str_c(st, co),
    cbsa_type = case_when(
      cbsa_kind == "Metropolitan Statistical Area" ~ "metro",
      cbsa_kind == "Micropolitan Statistical Area" ~ "micro"
    )
  )

stopifnot(nrow(delineation) == 1915, !anyDuplicated(delineation$delin_geoid),
          !anyNA(delineation$cbsa_type))

Since 2022 the Census Bureau uses Connecticut’s 9 planning regions instead of its 8 old counties (Federal Register notice of June 6, 2022), and the 2023 delineation lists only the regions. As in the research project, each old county gets the CBSA of its dominant planning region (the region that holds most of its people); the first table gives that region for each county, based on the Connecticut Office of Policy and Management’s (OPM) list of municipalities by planning region.

# Since 2022 Census uses Connecticut's 9 planning regions (codes 09110-09190)
# instead of its 8 old counties, and the 2023 delineation lists the regions.
# Our 2020 frame keeps the 8 counties, and a county can straddle regions.
# Each county takes the CBSA of its dominant planning region (the one with
# most of its population). The table gives that region's code, the county's
# number of towns, and how many of them lie in that region. Source: the
# Connecticut Office of Policy and Management (OPM) list of municipalities
# by planning region.
ct_counties <- tribble(
  ~GEOID,  ~delin_geoid, ~towns, ~towns_in_region,
  "09001", "09190",      23,     16,
  "09003", "09110",      29,     26,
  "09005", "09160",      26,     19,
  "09007", "09130",      15,     15,
  "09009", "09170",      27,     15,
  "09011", "09180",      21,     18,
  "09013", "09110",      13,     12,
  "09015", "09150",      15,     14
)

The second table lists, with one row per town, the towns of each county that lie in another planning region (34 of Connecticut’s 169 towns). The checks make sure every town is counted exactly once.

# The towns of each county that lie in another planning region, with that
# region's code (from the same OPM list of municipalities by planning region).
# A long list continues on a second row; separate_longer_delim() then makes
# one row per town.
ct_other_towns <- tribble(
  ~GEOID,  ~region, ~town,
  "09001", "09120", "Bridgeport, Easton, Fairfield, Monroe, Stratford, Trumbull",
  "09001", "09140", "Shelton",
  "09003", "09140", "Bristol",
  "09003", "09160", "Burlington, Hartland",
  "09005", "09140", "Bethlehem, Plymouth, Thomaston, Watertown, Woodbury",
  "09005", "09190", "Bridgewater, New Milford",
  "09009", "09140", "Ansonia, Beacon Falls, Cheshire, Derby, Middlebury, Naugatuck",
  "09009", "09140", "Oxford, Prospect, Seymour, Southbury, Waterbury, Wolcott",
  "09011", "09130", "Lyme, Old Lyme",
  "09011", "09150", "Voluntown",
  "09013", "09150", "Union",
  "09015", "09180", "Windham"
) |>
  separate_longer_delim(town, delim = ", ")

# Each of Connecticut's 169 towns is counted once: towns in the dominant
# region plus towns elsewhere equal the county's towns.
towns_elsewhere <- map_int(ct_counties$GEOID, \(g) sum(ct_other_towns$GEOID == g))
stopifnot(
  sum(ct_counties$towns) == 169, !anyDuplicated(ct_other_towns$town),
  all(ct_counties$towns == ct_counties$towns_in_region + towns_elsewhere),
  all(c(ct_counties$delin_geoid, ct_other_towns$region) %in% delineation$delin_geoid)
)

Look up the CBSA of each town’s planning region in the delineation and keep the towns whose CBSA differs from their county’s. These go into the ct_note column, so you can see where the county-level CBSA is only approximate.

# Look up the CBSA of every planning region in the delineation. Keep the
# towns whose region is in a different CBSA than their county's dominant
# region, and write them as "Town, Town (CBSA); Town (CBSA)".
region_cbsa <- delineation |>
  select(region = delin_geoid, cbsa_code, cbsa_title)
county_region_cbsa <- ct_counties |>
  left_join(region_cbsa, by = join_by(delin_geoid == region)) |>
  select(GEOID, county_cbsa = cbsa_code)

ct_notes <- ct_other_towns |>
  left_join(region_cbsa, by = "region") |>
  left_join(county_region_cbsa, by = "GEOID") |>
  filter(cbsa_code != county_cbsa) |>
  mutate(cbsa_name = str_remove(cbsa_title, ", CT$")) |>
  summarize(towns = str_c(town, collapse = ", "), .by = c(GEOID, cbsa_name)) |>
  summarize(towns_other_cbsa = str_c(towns, " (", cbsa_name, ")", collapse = "; "),
            .by = GEOID)

# Only Middlesex County (09007) lies wholly in one CBSA.
stopifnot(nrow(ct_notes) == 7, !"09007" %in% ct_notes$GEOID)
ct_notes

Join each county to its delineation row and write one row per county.

county_cbsa <- st_drop_geometry(counties_sf) |>
  select(GEOID, state_abbr, county_name) |>
  # Connecticut counties look up their dominant region; all others, themselves.
  left_join(select(ct_counties, GEOID, delin_geoid), by = "GEOID") |>
  mutate(delin_geoid = coalesce(delin_geoid, GEOID)) |>
  left_join(delineation |>
              select(delin_geoid, delin_name, cbsa_code, cbsa_title, cbsa_type,
                     cbsa_central, metdiv_code, metdiv_title, csa_code, csa_title),
            by = "delin_geoid") |>
  left_join(ct_counties, by = c("GEOID", "delin_geoid")) |>
  left_join(ct_notes, by = "GEOID") |>
  mutate(
    cbsa_ct_approx = state_abbr == "CT",
    ct_note = if_else(
      cbsa_ct_approx,
      str_c("Via ", delin_name, " (", towns_in_region, " of ", towns, " towns). ",
            "Towns in another CBSA: ", replace_na(towns_other_cbsa, "none"), "."),
      NA_character_
    )
  ) |>
  select(GEOID, state_abbr, county_name, delin_geoid, cbsa_code, cbsa_title,
         cbsa_type, cbsa_central, metdiv_code, metdiv_title, csa_code, csa_title,
         cbsa_ct_approx, ct_note) |>
  arrange(GEOID)

# Grain: one row per frame county. A missing cbsa_code means the county lies
# outside every metro and micro area (not a missing value).
stopifnot(
  nrow(county_cbsa) == 3108, !anyDuplicated(county_cbsa$GEOID),
  setequal(county_cbsa$GEOID, counties_sf$GEOID),
  !anyNA(county_cbsa$cbsa_code[county_cbsa$state_abbr == "CT"]),
  all(is.na(county_cbsa$cbsa_code) == is.na(county_cbsa$cbsa_type))
)

write_csv(county_cbsa, file.path(clean_dir, "county_cbsa_2023.csv"), na = "")
count(county_cbsa, cbsa_type)

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
counties_2020.csv	3,108	a contiguous-U.S. county or DC	0.33 MB
counties_2020.gpkg	3,108	the same county, with its outline (layer counties_2020, EPSG:4326)	17.7 MB
census_bg_centers_2020.csv	238,193	a contiguous-U.S. block group	10.2 MB
us_states_2020.csv	51	a state or DC	0.004 MB
us_states_2020.gpkg	51	the same state, with its simplified outline (layer us_states_2020, EPSG:4326)	0.41 MB
canada_provinces_2021.csv	13	a province or territory	0.001 MB
canada_provinces_2021.gpkg	13	the same province, with its simplified outline (layer canada_provinces_2021, EPSG:4326)	1.13 MB
county_cbsa_2023.csv	3,108	a contiguous-U.S. county with its metro or micro area	0.25 MB

Clean variables

counties_2020.csv and counties_2020.gpkg share these columns; the .gpkg adds the geometry column geom. Rows are sorted by GEOID, and no cell is missing.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (state + county), 2020 boundaries
county_name	chr	—	County name with its legal description, e.g., Monroe County, Orleans Parish, Richmond city (raw NAMELSAD)
STATEFP	chr	—	2-digit state FIPS code
COUNTYFP	chr	—	3-digit county FIPS code within the state
state_abbr	chr	—	2-letter postal abbreviation
state_name	chr	—	State name
census_region	chr	—	Census region: Northeast, Midwest, South, West
census_division	chr	—	One of the 9 Census divisions, e.g., Middle Atlantic
east_split	chr	—	The research project’s eastern split, used in its descriptive tables of Canadian smoke (project scripts 05a_counties_seer.R and 06_episode_count.R): New England (CT ME MA NH RI VT), Mid-Atlantic (DE DC MD NJ NY PA), Great Lakes (IL IN MI OH WI), Upper Midwest (IA MN ND SD), Other (all remaining states)
land_km2	dbl	km²	Land area (raw ALAND / 1,000,000), 2 decimals
water_km2	dbl	km²	Water area inside the legal county boundary (raw AWATER / 1,000,000), including Great Lakes and coastal water
lat	dbl	degrees north	Latitude of the county’s population-weighted center: mean of block-group center latitudes weighted by population
lon	dbl	degrees east (negative = west)	Longitude of the population-weighted center, weighted by population × cos(latitude)
pop_2020	dbl	persons	2020 Census population (April 1, 2020), the sum of the county’s block groups; a whole number
geom (.gpkg only)	MULTIPOLYGON	EPSG:4326	County outline from the 2020 cartographic file, clipped to the shoreline, not simplified further

census_bg_centers_2020.csv: one row per block group, sorted by bg_geoid, no missing cells.

Variable	Type	Units	Description
bg_geoid	chr	—	12-character block-group code: state (2) + county (3) + tract (6) + block group (1)
GEOID	chr	—	5-character county code (first 5 characters of bg_geoid); joins to counties_2020.csv
pop_2020	dbl	persons	2020 Census population; 0 for 1,310 block groups (zero weight in any average)
lat	dbl	degrees north	Latitude of the block group’s center of population, as published
lon	dbl	degrees east (negative = west)	Longitude of the center of population, as published

us_states_2020.csv and us_states_2020.gpkg share these columns; the .gpkg adds geom. Rows are sorted by STATEFP, and no cell is missing.

Variable	Type	Units	Description
STATEFP	chr	—	2-digit state FIPS code
state_abbr	chr	—	2-letter postal abbreviation
state_name	chr	—	State name
census_region	chr	—	Census region
census_division	chr	—	Census division
east_split	chr	—	Project grouping, as in counties_2020.csv
contiguous	lgl	—	TRUE for the 48 contiguous states and DC; FALSE for Alaska and Hawaii
land_km2	dbl	km²	Land area from the TIGER/Line state file
water_km2	dbl	km²	Water area from the TIGER/Line state file
pop_2020	dbl	persons	2020 Census population, the sum of the state’s block groups
geom (.gpkg only)	MULTIPOLYGON	EPSG:4326	State outline simplified with a 500 m tolerance; includes coastal water and the state’s share of the Great Lakes, out to the international border

canada_provinces_2021.csv and canada_provinces_2021.gpkg share these columns; the .gpkg adds geom. Rows are sorted by PRUID, and no cell is missing.

Variable	Type	Units	Description
PRUID	chr	—	2-digit province or territory code (raw PRUID); read it with col_types = cols(PRUID = "c")
prov_abbr	chr	—	2-letter postal abbreviation: NL, PE, NS, NB, QC, ON, MB, SK, AB, BC, YT, NT, NU
prov_name	chr	—	English name (raw PRENAME)
prov_name_fr	chr	—	French name (raw PRFNAME)
land_km2	dbl	km²	Land area published by Statistics Canada (raw LANDAREA)
geom (.gpkg only)	MULTIPOLYGON	EPSG:4326	Outline simplified with a 1 km tolerance; islands under 1 km² dropped. Inland lakes such as Lake Winnipeg stay inside; the Great Lakes and the St. Lawrence are erased as coastal water, so they lie in no province. Small enclosed holes are filled

county_cbsa_2023.csv: one row per county in counties_2020.csv, sorted by GEOID. Read the code columns as text: col_types = cols(cbsa_code = "c", metdiv_code = "c", csa_code = "c").

Variable	Type	Units	Description
GEOID	chr	—	5-character county code
state_abbr	chr	—	2-letter postal abbreviation
county_name	chr	—	County name, as in counties_2020.csv
delin_geoid	chr	—	Code looked up in the delineation: equal to GEOID, except in Connecticut, where it is the dominant planning region (09110–09190)
cbsa_code	chr	—	5-digit CBSA code, July 2023. Empty = the county is in no metro or micro area (a real answer, not missing data)
cbsa_title	chr	—	CBSA name; empty outside a CBSA
cbsa_type	chr	—	metro (metropolitan) or micro (micropolitan); empty outside a CBSA
cbsa_central	chr	—	Central or Outlying county of its CBSA (for Connecticut, the planning region’s status); empty outside a CBSA
metdiv_code	chr	—	5-digit metropolitan division code; empty when the CBSA has no divisions
metdiv_title	chr	—	Metropolitan division name; empty when none
csa_code	chr	—	3-digit CSA code; empty when the CBSA belongs to no CSA
csa_title	chr	—	CSA name; empty when none
cbsa_ct_approx	lgl	—	TRUE for the 8 Connecticut counties, whose CBSA comes from the dominant planning region and is therefore approximate; FALSE elsewhere
ct_note	chr	—	Connecticut only: the planning region used, its share of the county’s towns, and the towns that lie in another CBSA; empty elsewhere

Checks and cautions

• County frame matches the project. counties_2020.csv has the same 3,108 GEOIDs as the project’s counties_2020.parquet (built from the TIGER/Line county file). County names, state codes and names, Census regions and divisions, and east_split agree for all 3,108 rows. Land and water areas agree within 0.005 km² (rounding). east_split counts: Great Lakes 437, Mid-Atlantic 178, New England 67, Upper Midwest 305, Other 2,121.
• Population totals are exact. County populations sum to 329,260,619, which is the official 2020 Census count for the 50 states and DC (331,449,281) minus Alaska (733,391) and Hawaii (1,455,271). The state file sums to 331,449,281, and each contiguous state equals the sum of its counties. The research project also uses July 1, 2020 county population estimates from SEER, a National Cancer Institute program that republishes Census Bureau estimates (we used only its population counts, and the file is not in the student bundle). Against those estimates, the total differs by 0.04% and the median county by 0.18%. Census counts and estimates measure slightly different things, so do not mix them in one analysis.
• Population centers. 3,095 of the 3,108 population-weighted centers lie inside their own county. The other 13 are expected: the point marks where people live on average, not a spot inside the county. Five fall in water (Monroe County, Florida, in the Keys; Hancock County, Maine; Perquimans County, North Carolina; Island and Pacific Counties, Washington), six fall in a city that the county surrounds (Baltimore County, Maryland, and five Virginia counties around independent cities), and two fall in a neighboring county across a river (Lee County, Iowa; Pepin County, Wisconsin). The median distance from the Census internal point (a point inside the county) is 5.7 km.
• Block groups. 238,193 rows, one per block group, all with a 12-character bg_geoid that starts with a county GEOID in the frame. 1,310 have zero population and are kept with weight 0.
• 2020 Census noise. The Census Bureau added small amounts of random noise to 2020 counts below the state level to protect privacy. County totals are reliable; counts for single block groups are less precise.
• State outlines. Simplification cut the state file from 967,124 to 17,524 vertices (the corner points that make up an outline), and every simplified state keeps its area within 0.7% of the original. No two states overlap. As a test, 99.992% of the 238,193 block-group centers fall in their own state. Of the other 19, nine already lie outside their own state in the full-detail file (an average point can sit across a river border), and ten lie within 340 m of a state line, where simplification moved the border. State outlines include coastal water and the Great Lakes, while county outlines stop at the shore, so the two layers do not line up along coasts. Use the county layer for land maps.
• Province outlines. The raw file has 175,779 separate pieces of land. Nunavut has 36% of them; Ontario, British Columbia, Quebec, and Newfoundland and Labrador have most of the rest. The script dropped the 172,159 pieces (islands) smaller than 1 km² (about 4,850 km² in total, 0.05% of Canada). Water enclosed by a single province (91 holes in the pieces of land that were kept, about 370 km² in total, mostly in Nunavut and Ontario) was filled and now counts as part of that province. The simplified layer has 64,433 vertices, no holes, and no overlaps between provinces. As a test, 78,157 random points on the full-detail land were labelled with the simplified layer: 99.67% got the right province; 0.30% (236 points) fell in no province, mostly near coasts (median 208 m from the nearest province; the farthest, 16 km, sat on a dropped island); and 0.03% (24 points, all within 790 m of a border) got a neighbor. When you label fire detections, give points that fall in no polygon the nearest province with st_nearest_feature(): in the test, this gave the right province for all 236 such points.
• Province area. Province polygons include inland lakes such as Lake Winnipeg and Great Slave Lake, so their area is larger than land_km2 (Quebec: about 1,502,000 km² of polygon, 1,298,600 km² of land). Use land_km2 for land area.
• U.S. and Canadian layers do not share a border. The state and province layers come from two agencies and were simplified separately (500 m and 1 km). Along the international border they overlap in 18 state–province pairs, 318 km² in total (the largest are Alaska–Yukon, 112 km², and Alaska–British Columbia, 37 km²), and elsewhere they leave thin gaps. In a test with 63,625 random land points within 5 km of the border, about 0.4% fell in both layers, and 3.8% of the Canadian points (0.5% of the U.S. points) fell in neither. A point outside its own country’s layer was usually close to it (median 160 m; 99% within 1.4 km). The farthest, up to 4.3 km, sat on small islands that the script dropped (Gulf Islands, Thousand Islands, Rainy Lake, Lake of the Woods).
• Border lakes and rivers. The two agencies treat water differently. U.S. states include their half of the Great Lakes and of border rivers, out to the international line. Statistics Canada treats the Great Lakes and the St. Lawrence as coastal water and erases them, so their Canadian part lies in no polygon at all. Test points in Georgian Bay, on the Canadian side of Lakes Superior, Huron, Erie, Ontario, and St. Clair, and in the St. Lawrence (Lake Saint-Pierre and the estuary) fall in no state and no province. Other lakes are inside their province: the Canadian part of Lake of the Woods (a border lake) is in Ontario, and Lake Nipigon and Lake Winnipeg are in Ontario and Manitoba.
• How to label fire detections near the border. Take each detection’s country from the FIRMS file it came from (the yearly _Canada or _United_States file), label it only with that country’s layer, and give a point that falls in no polygon the nearest state or province with st_nearest_feature(). In the test above, this rule gave the right state or province for 63,622 of 63,625 points (99.995%); the 3 misses lie on a state line (Montana–North Dakota, North Dakota–Minnesota) right at the border. Testing each point against both layers instead would give some border points two labels and leave others with none.
• Metro areas match the project. county_cbsa_2023.csv agrees with the project’s county_cbsa_2023.parquet in every column for all 3,108 counties (0 mismatches). 1,833 counties are in a CBSA (1,179 metro, 654 micro) and 1,275 are in none. These hold 94.6% of the 2020 population (metro 86.2%, micro 8.4%). 139 counties are in a metropolitan division and 1,269 in a CSA.
• Connecticut is approximate. In all 8 Connecticut counties, the dominant planning region has both the most towns and a majority of the 2020 population: from about 55% in Litchfield County to 100% in Middlesex County. We calculated these shares from 2020 town populations, which are not in the bundle. For 7 of the 8 Connecticut counties, some towns lie in a different CBSA than the one assigned (see ct_note). The Waterbury-Shelton CBSA (47930) has no county in this file: Waterbury, in New Haven County, is assigned to the New Haven CBSA. Any source that uses Connecticut planning regions (codes 09110–09190, used from 2022 on) must be mapped back to the 8 counties before joining. The town lists typed into chunk m00-ct-towns agree town by town with the research project’s lists and with the OPM list of municipalities by planning region (all 34 towns, and the number of towns in each of the 9 regions). The CBSA of each town’s planning region is looked up in the delineation file, not typed in.
• Reading the files. read_csv() keeps codes with a leading zero (GEOID, STATEFP, bg_geoid) as text, but it turns PRUID, cbsa_code, metdiv_code, and csa_code into numbers unless you give col_types. It reads every whole number, such as pop_2020, as dbl.
• Writing GeoPackages on external or synced drives. A GeoPackage is a small SQLite database (SQLite is a database stored in a single file). On some drives, such as an exFAT external drive or a cloud-synced folder, SQLite can refuse the last step of a write. GDAL then prints “attempt to write a readonly database”. We built the bundle on an exFAT drive, where 2 of 8 test writes of counties_2020.gpkg hit this. The rows were all there, but the file missed one of its internal triggers (rules that keep the stored row count up to date). Running Sys.setenv(OGR_SQLITE_JOURNAL = "MEMORY") before the write chunks fixed it (0 failures in 24 test writes). Writing to a folder on your computer’s main drive also avoids it.
• Memory and time. The whole script runs in about 25 seconds. Its peak memory use is about 2 GB (1.7 to 2.2 GB in our runs), so close other large R sessions first. Most of the peak comes from simplifying the full-detail state outlines in m00-write-states.

SEER county population

The Surveillance, Epidemiology, and End Results (SEER) program of the National Cancer Institute (NCI) publishes population estimates for every U.S. county and every year from 1969 to 2024. Each number estimates how many people lived in a county on July 1 of that year, by race, sex, and single year of age. The estimates come from the Census Bureau’s population estimates, which start from the last census and add births, subtract deaths, and add people who moved in minus people who moved out. SEER adjusts a few county codes and areas so that each county means the same place in every year.

In this project these counts are the denominators: they turn county totals into per-person numbers and give population weights, for example to compute the share of U.S. residents who had a smoke day. The age columns let you describe exposure for children and for adults 65 and older. They are model estimates for July 1, not census counts, and they cannot tell you where people actually were on a given day (commuting, travel, or seasonal residents).

Provider	National Cancer Institute, SEER Program, from Census Bureau population estimates (2010–2019 years made for NCI by Woods & Poole Economics, Inc.)
Coverage	raw: 1969–2024, clean: 2006–2024 · raw: 50 states and DC; clean: the 3,108 counties of the contiguous U.S. and DC (2020 boundaries)
Raw unit	one row = the July 1 population of one county × year × race × sex × single year of age
Clean unit	one row = a county × year (3,108 counties × 19 years)
Licence and citation	Free public download; the provider states no licence terms. Cite: National Cancer Institute, SEER Program, U.S. County Population Data, 1969–2024 (single ages through 89 and 90+, Katrina/Rita adjusted), released February 2026, https://seer.cancer.gov/popdata/. SEER’s suggested citation for the 2010–2019 years names the Woods & Poole Economics intercensal estimates (revised estimates for the years between two censuses) by age, sex, and bridged race (January 2026)
Documentation	SEER county population data; data dictionary (copy in raw/seer/); county code modifications; single-year-of-age methods; download page

Raw files

File (in wildfire_data/raw/)	Size	Format	One row is
seer/us.1969_2024.singleages.through89.90plus.adjusted.txt.gz	284.1 MB	gzip-compressed fixed-width ASCII text: 26 characters per line, no header, no separators, Windows (CR/LF) line ends; 69,655,933 lines, about 2 GB unzipped	the July 1 population of one county × year × race × sex × single year of age
seer/popdic__dl20260927.html	0.03 MB	HTML web page: a copy of SEER’s data dictionary, downloaded 2026-09-27	not a data file: it gives the start column, length, and codes of each field

Raw variables

The data file us.1969_2024.singleages.through89.90plus.adjusted.txt.gz has no column names. Each field sits at fixed character positions (a fixed-width file), as listed in the data dictionary popdic__dl20260927.html. The names in the first column are the dictionary’s names. The code values below are the dictionary’s; we checked which ones occur in this file by reading every line.

Variable	Type	Description
Year	numeric, columns 1–4	Year of the estimate, 1969–2024. Each estimate is for July 1 of that year
State postal abbreviation	character, columns 5–6	Two-letter postal code, e.g., NY. KR is a dummy state SEER created for Hurricane Katrina/Rita evacuees; it appears only in 2005 (540 lines)
State FIPS code	numeric, columns 7–8	2-digit state FIPS code (Federal Information Processing Standards: the federal numeric code of a state), zero-filled, e.g., 36 = New York; 99 (9-filled) for the KR dummy state
County FIPS code	numeric, columns 9–11	3-digit county code within the state, e.g., 055; 999 for the KR dummy state. Columns 7–11 together form the 5-character county code. In 2006–2024, two codes in the contiguous U.S. differ from the 2020 Census: 46113 (Oglala Lakota County, SD, called Shannon County until 2015; Census code 46102) and 51917 (Bedford County, VA, combined with Bedford city, 51515, in all years; Census code 51019). Alaska also uses older codes that differ from the 2020 Census (02261 and 02270 in all of 2006–2024; 02201, 02232, and 02280 in 2006–2013), but Alaska is dropped
(columns 12–13)	blank	Two spaces in every line; not documented by the provider
Race	numeric, column 14	Bridged race. Since 2000 the Census lets people report more than one race; “bridged race” assigns each person to one race so that the series stays comparable over time. This file is SEER’s “1969+” White/Black/Other file: 1 White, 2 Black, 3 Other. The dictionary does not define “Other” further. In the dictionary’s 1990+ codes, the groups other than White and Black are 3 American Indian/Alaska Native and 4 Asian or Pacific Islander; those codes belong to other SEER files and do not occur here
Origin	numeric, column 15	Hispanic origin: 0 non-Hispanic, 1 Hispanic, 9 not applicable in the 1969+ White/Black/Other files. Every line of this file has 9, so it has no Hispanic-origin detail
Sex	numeric, column 16	1 male, 2 female
Age	numeric, columns 17–18	Single year of age: 00–89 = age in years, 90 = 90 and older. All years in this file use these codes. (The dictionary also lists codes 00–19 for SEER’s age-group files: 00 = 0 years, 01 = 1–4, 02 = 5–9, … , 19 = 90+; they are not used here)
Population	numeric, columns 19–26	Estimated number of people on July 1, zero-filled to 8 digits (e.g., 00000159 = 159). Groups with no people are left out: no line has a population of 0

The dictionary page popdic__dl20260927.html has no variables of its own; it is the documentation for the table above.

Cleaning steps

The code below makes these decisions:

1. Stream the file instead of loading it. Unzipped, the file is about 2 GB, and the full table would need several GB of memory. The code reads 1 million lines at a time through a connection (an open file that R reads step by step) and keeps only a small summary of each piece. Pieces from before 2006 are skipped without being parsed. The run takes about 2 minutes and about 1 GB of memory. readr::read_lines() and read_lines_chunked() are not used because, for a .gz file, both first unzip the whole file (about 2 GB) to a temporary file on disk.
2. Cut columns by position. str_sub() takes the characters at the positions given in the data dictionary.
3. Sum over race and origin, and group the ages. The clean file is one row per county and year, so race and origin are summed away, and single ages are grouped into 0–4, 5–17, 18–64, 65–74, 75–84, and 85+. These six groups add up to every age column of the clean file.
4. Keep the contiguous U.S. Alaska and Hawaii are dropped. The KR dummy state exists only in 2005, but the filter would drop it too.
5. Recode two SEER county codes. 46113 becomes 46102 (Oglala Lakota County, SD) and 51917 becomes 51019 (Bedford County, VA). In both cases the SEER area is the same land as the 2020 county.
6. Complete panel. Every county gets every year. A missing county-year would stop the script, because it would be an error, not a zero. A missing sex-and-age group inside a county-year is a true zero, because SEER leaves out groups with no people.
7. Check the download. The script counts the lines it read. A file that stopped downloading early is still read, with only a warning, so the line count is the check.

The first chunk names the raw file, sets the years to keep, and reads the county list written by module m00.

dir.create(clean_dir, recursive = TRUE, showWarnings = FALSE)

seer_file <- file.path(raw_dir, "seer",
                       "us.1969_2024.singleages.through89.90plus.adjusted.txt.gz")
# The years we keep. The file starts in 1969; this module keeps 2006-2024.
years <- 2006:2024

# The county universe: 3,108 contiguous-U.S. counties plus DC, 2020 borders.
counties <- read_csv(file.path(clean_dir, "counties_2020.csv"),
                     col_types = cols_only(GEOID = col_character(),
                                           county_name = col_character(),
                                           state_abbr = col_character()))
stopifnot(nrow(counties) == 3108, !anyDuplicated(counties$GEOID))

Before writing any parsing code, look at a few raw lines. gzfile() opens the compressed file, and readLines(..., n = 5) reads only its first 5 lines.

# readLines() on a gzfile() connection unzips only what it reads, so this
# looks at the first 5 lines without touching the other 70 million.
first_lines <- readLines(gzfile(seer_file), n = 5)
first_lines
# Every line has 26 characters and no separators: a fixed-width record.
nchar(first_lines)

In the first line, 1969AL01001 1910000000159, you can read the year (1969), the state (AL), the county code (01001), two blanks, race 1, origin 9, sex 1, age 00, and population 00000159. The next function cuts many lines at once with str_sub(x, start, end), which takes characters start to end of every string in x.

# Cut fixed-width lines into columns. Positions come from the SEER record
# layout in raw/seer/popdic__dl20260927.html. Columns 12-13 are blank in
# this file, so we skip them.
parse_seer_lines <- function(lines) {
  tibble(
    year       = as.integer(str_sub(lines, 1, 4)),
    state_abbr = str_sub(lines, 5, 6),
    seer_code  = str_sub(lines, 7, 11),       # state FIPS (2) + county FIPS (3)
    race       = as.integer(str_sub(lines, 14, 14)),
    origin     = as.integer(str_sub(lines, 15, 15)),
    sex        = as.integer(str_sub(lines, 16, 16)),
    age        = as.integer(str_sub(lines, 17, 18)),   # 0-89 = years, 90 = 90+
    pop        = as.numeric(str_sub(lines, 19, 26))
  )
}

parse_seer_lines(first_lines)

The next function summarises one piece of the file. It first keeps only the lines from 2006–2024, by looking at the year text, and only then parses them, because parsing is the slow step.

# What we keep from one piece of the file: population summed over race and
# origin, by year x county x sex x age group. Six age groups are enough to
# build every age column of the clean file.
summarise_seer_chunk <- function(lines, years) {
  # Filter on the year text first: cutting a million lines into eight
  # columns is the slow step, so only do it for the lines we keep.
  lines <- lines[as.integer(str_sub(lines, 1, 4)) %in% years]
  if (length(lines) == 0) return(NULL)

  parse_seer_lines(lines) |>
    mutate(age_group = case_when(
      age < 5  ~ "0-4",
      age < 18 ~ "5-17",
      age < 65 ~ "18-64",
      age < 75 ~ "65-74",
      age < 85 ~ "75-84",
      .default = "85+"
    )) |>
    summarise(pop = sum(pop), .by = c(year, state_abbr, seer_code, sex, age_group))
}

The loop reads the file one piece at a time and stores each summary in pieces, a list made in advance with room for 100 summaries, so the loop fills it instead of growing it. next jumps to the next piece, and break leaves the loop at the end of the file.

# The unzipped file is about 2 GB (69.7 million lines), too big to load
# at once. We read 1 million lines at a time, keep a small summary of each
# piece, and throw the lines away before reading the next piece.
# (readr's read_lines() and read_lines_chunked() are not used: for a .gz
# file, both first unzip the whole file to a temporary file on disk.)
# This takes about 2 minutes and about 1 GB of memory.
con <- gzfile(seer_file, open = "rt")
pieces <- vector("list", 100)   # room for 100 pieces; the file needs 70
n_pieces <- 0
n_lines <- 0
repeat {
  lines <- readLines(con, n = 1e6)
  if (length(lines) == 0) break             # end of the file
  n_pieces <- n_pieces + 1
  n_lines <- n_lines + length(lines)
  # The file is sorted by year, so if the LAST line of a piece is from
  # before 2006, the whole piece is: skip it without parsing.
  last_year <- as.integer(str_sub(lines[length(lines)], 1, 4))
  if (n_pieces %% 10 == 0) message("Piece ", n_pieces, ": read through ", last_year)
  if (last_year < min(years)) next
  pieces[[n_pieces]] <- summarise_seer_chunk(lines, years)
}
close(con)
rm(lines)

# A download that stopped early is still read, with only a warning and
# fewer lines. This release has exactly 69,655,933 lines; if the check
# fails, download the file again.
n_lines
stopifnot(n_lines == 69655933, last_year == 2024)

A county-year can start at the end of one piece and continue in the next, so the summaries are combined and summed again. Each year has about 37,700 rows: about 3,140 county codes (including Alaska and Hawaii) × 2 sexes × 6 age groups.

# One county-year can be split across two pieces, so sum the pieces again.
seer_groups <- bind_rows(pieces) |>
  summarise(pop = sum(pop), .by = c(year, state_abbr, seer_code, sex, age_group))
rm(pieces)

# Grain: one row per year x SEER county code x sex x age group.
stopifnot(!anyDuplicated(select(seer_groups, year, seer_code, sex, age_group)),
          all(seer_groups$year %in% years), all(seer_groups$pop >= 0))
seer_groups |> count(year)

Next, drop the states outside the contiguous U.S. and give SEER’s two special codes their 2020 Census codes. The why column documents each recode; it is not used in the join.

# SEER keeps two county codes that differ from the 2020 Census codes. In
# both cases the SEER area is the same land as the 2020 county.
seer_recode <- tribble(
  ~seer_code, ~GEOID,  ~why,
  "46113",    "46102", "Oglala Lakota County, SD, was Shannon County (46113) until 2015",
  "51917",    "51019", "Bedford County, VA, plus Bedford city (51515), which joined it in 2013"
)

seer_county <- seer_groups |>
  # Alaska and Hawaii are outside the contiguous U.S. "KR" is SEER's dummy
  # state for 2005 hurricane evacuees; it does not appear after 2005.
  filter(!state_abbr %in% c("AK", "HI", "KR")) |>
  left_join(select(seer_recode, seer_code, GEOID), by = "seer_code") |>
  mutate(GEOID = coalesce(GEOID, seer_code))

# Every remaining SEER code must now be a county in our universe. Other
# years have more combined codes (see the guide), so this check stops the
# script if you change `years` and meet one of them.
stopifnot(all(seer_county$GEOID %in% counties$GEOID))

sum(pop[condition]) adds up only the rows where the condition is true, so one summarise() builds all ten population columns.

# Sum over sex and age groups to get one row per county and year. SEER
# leaves out cells with zero people, so a group with no rows sums to 0,
# which is a true zero.
seer_wide <- seer_county |>
  summarise(
    pop_total   = sum(pop),
    pop_female  = sum(pop[sex == 2]),
    pop_male    = sum(pop[sex == 1]),
    pop_under5  = sum(pop[age_group == "0-4"]),
    pop_under18 = sum(pop[age_group %in% c("0-4", "5-17")]),
    pop_lt65    = sum(pop[age_group %in% c("0-4", "5-17", "18-64")]),
    pop_65p     = sum(pop[age_group %in% c("65-74", "75-84", "85+")]),
    pop_65_74   = sum(pop[age_group == "65-74"]),
    pop_75_84   = sum(pop[age_group == "75-84"]),
    pop_85p     = sum(pop[age_group == "85+"]),
    .by = c(GEOID, year)
  )

Now attach the populations to the full county × year grid and check that nothing is missing. The last lines list the counties with an older age group of zero people.

# Start from every county x year and attach the populations. A county-year
# with no SEER rows would be an error, not a zero, so we check for NA.
seer_county_year <- expand_grid(GEOID = counties$GEOID, year = years) |>
  left_join(seer_wide, by = c("GEOID", "year")) |>
  arrange(GEOID, year)

stopifnot(nrow(seer_county_year) == 3108 * length(years),
          !anyNA(seer_county_year),
          nrow(seer_wide) == nrow(seer_county_year))

# Small counties can have no one in an older age group in some years.
seer_county_year |>
  filter(pop_65_74 == 0 | pop_75_84 == 0 | pop_85p == 0) |>
  left_join(counties, by = "GEOID") |>
  count(GEOID, county_name, state_abbr, name = "years_with_a_zero_age_group")

Before writing, check the grain (one row per county and year) and the identities that must hold between the columns: females + males = total, under 65 + 65 and older = total, and so on.

# Grain: one row per county (GEOID) x year, 3,108 x 19 = 59,052 rows.
pop_cols <- names(seer_county_year)[str_starts(names(seer_county_year), "pop_")]
stopifnot(
  !anyDuplicated(select(seer_county_year, GEOID, year)),
  all(nchar(seer_county_year$GEOID) == 5),
  all(as.matrix(seer_county_year[pop_cols]) >= 0),
  with(seer_county_year, all(pop_total == pop_female + pop_male)),
  with(seer_county_year, all(pop_total == pop_lt65 + pop_65p)),
  with(seer_county_year, all(pop_65p == pop_65_74 + pop_75_84 + pop_85p)),
  with(seer_county_year, all(pop_under5 <= pop_under18 & pop_under18 <= pop_lt65)),
  all(seer_county_year$pop_total > 0)
)
write_csv(seer_county_year, file.path(clean_dir, "seer_county_year_2006_2024.csv"),
          na = "")

A last look at the result: the contiguous U.S. grows from about 296 million people in 2006 to 338 million in 2024, and the share aged 65 and older rises from 12.5% to 18.0%.

# A quick look: total population and the share aged 65+ in the contiguous
# U.S. The 65+ share rises every year as the baby boom generation ages.
seer_county_year |>
  summarise(pop_total = sum(pop_total), pop_65p = sum(pop_65p), .by = year) |>
  mutate(share_65p = round(100 * pop_65p / pop_total, 1))

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
seer_county_year_2006_2024.csv	59,052	a county × year (3,108 counties × 19 years, July 1 populations)	3.74 MB

Clean variables

seer_county_year_2006_2024.csv: one row per county and year, sorted by GEOID and year. No cell is missing, and every population is a whole number.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (2-digit state + 3-digit county), 2020 boundaries; matches counties_2020.csv
year	dbl	year	Year of the estimate (2006–2024); the population is for July 1 of that year
pop_total	dbl	persons	All ages, both sexes
pop_female	dbl	persons	Females, all ages (SEER sex code 2)
pop_male	dbl	persons	Males, all ages (SEER sex code 1); pop_female + pop_male = pop_total
pop_under5	dbl	persons	Ages 0–4
pop_under18	dbl	persons	Ages 0–17 (includes pop_under5)
pop_lt65	dbl	persons	Ages 0–64 (includes pop_under18)
pop_65p	dbl	persons	Ages 65 and older; pop_lt65 + pop_65p = pop_total and pop_65p = pop_65_74 + pop_75_84 + pop_85p
pop_65_74	dbl	persons	Ages 65–74
pop_75_84	dbl	persons	Ages 75–84
pop_85p	dbl	persons	Ages 85 and older (SEER single ages 85–89 plus the 90+ group)

Checks and cautions

• Same numbers as the research project. Our validation (an instructor-side check against the research project’s cleaned file, seer_county_year.parquet) found the same numbers. The project file was built with a different method (a shell awk program). For all 59,052 county-years, pop_total, pop_lt65, pop_65p, pop_65_74, pop_75_84, and pop_85p are identical (0 differences). The project file has no sex, under-5, or under-18 columns, so the check also rebuilt 2006, 2013, and 2024 a third way (gzip and grep to pick the years, then read_fwf(), then sums over single ages). All 10 population columns agree for all 9,324 county-years (0 differences).
• What is in the raw file. We read every line: 69,655,933 lines, 26,730,643 of them from 2006–2024. Every line has 26 characters. Race codes are 1–3 only, origin is always 9, sex is 1 or 2, age runs 00–90, columns 12–13 are always blank, and no line has a population of 0.
• Every county, every year. All 3,108 counties have all 19 years, with no missing values. In 2006–2024, SEER uses only two county codes in the contiguous U.S. that differ from the 2020 Census (46113, 51917; see Cleaning steps). Alaska has five more (02201, 02232, 02261, 02270, 02280), but Alaska is not in the clean file. The bundle copy of the raw file is byte-identical to the project’s.
• Zeros are real. Loving County, Texas (48301), is the smallest county (46 to 82 people). It has pop_85p = 0 in all 19 years, pop_75_84 = 0 in 10 years, and pop_under5 = 0 in 2 years. No other county has a zero in any column. Do not divide by an age group without checking for zeros.
• Estimates, not census counts. Compared with the 2020 Census counts (April 1, 2020) in counties_2020.csv, the SEER July 1, 2020 estimates total 329,393,451 people against 329,260,619 (0.04% more). The median county differs by 0.18%, and 99% of counties differ by less than 1.7%. The largest gaps are Luce County, Michigan (+20.1%), Hampshire County, Massachusetts (−8.5%), Tattnall County, Georgia (+6.5%), and Tompkins County, New York (−5.6%). Hampshire and Tompkins have large college populations. Use one source consistently in an analysis; do not mix census counts and estimates.
• Different producers for different years. SEER’s 2006–2009 estimates come from the Census Bureau, the 2010–2019 estimates were made for NCI by Woods & Poole Economics, and the 2020–2024 estimates are the Census Bureau’s Vintage 2024 estimates (a vintage is the year a series of estimates was released; each new vintage revises earlier years). Numbers on the Census website can therefore differ slightly from these.
• Connecticut. For 2020–2024, the Census Bureau’s estimates use Connecticut’s 9 planning regions. Woods & Poole converted them back to the 8 old counties for SEER, using population shares. Connecticut county values for these years are therefore modeled, not published Census numbers.
• Small groups are uncertain. SEER warns that county estimates by single year of age have limited reliability. The clean file uses broad age groups, but pop_under5 and the oldest groups in small counties are still rough.
• Race and Hispanic origin. The clean file sums over race. This raw file has only three race groups (White, Black, Other) and no Hispanic-origin detail.
• Katrina/Rita adjustment. The “adjusted” in the file name refers to SEER’s changes for 2005 in Alabama, Louisiana, Mississippi, and Texas, after hurricanes Katrina and Rita. It does not affect 2006–2024.
• Other years. If you change years to include years before 2006, the check at the end of chunk m06-recode stops the script. According to the research project’s build, SEER uses more combined or changed codes in earlier years, for example 08911–08914 (Adams, Boulder, Jefferson, and Weld counties, Colorado, each with part of Broomfield, 1990–2001) and 04910 (Yuma and La Paz counties, Arizona, 1990–1993). Some of these cannot be split into 2020 counties.
• Reading the file. With no col_types, read_csv() reads GEOID as chr, but year and every population as dbl. Use col_types = cols(GEOID = col_character(), year = col_integer(), .default = col_double()) if you want year as an integer.
• Memory and time. On our computer (a 10-core Mac) the code ran in 100 to 220 seconds (the longer times while other programs were busy), with a peak memory use of 0.9 to 1.1 GB.

NASA FIRMS active fires (MODIS)

MODIS (Moderate Resolution Imaging Spectroradiometer) is a sensor on two NASA satellites, Terra and Aqua. Between them they observe almost every place at least four times a day: Terra crosses the equator at about 10:30 and 22:30 local time, and Aqua at about 13:30 and 01:30. An algorithm flags each 1 km pixel that is much hotter than its surroundings as an active fire detection and records its location, time, a confidence score, and its fire radiative power (FRP): the heat the fire gives off as radiation at that moment, in megawatts (MW). NASA’s Fire Information for Resource Management System (FIRMS) publishes these detections; this bundle has its yearly files for Canada, the United States, and Mexico, 2003–2024.

In this project the detections are the “source” side of the smoke story: where Canadian fires burned, on which days, and how hard. FRP summed over a day is a rough measure of how much smoke a region’s fires could produce, and Canada’s 2023 season stands far above every other year. The detections cannot tell you how much smoke was made, where it went, or how much land burned. A satellite sees only what burns during its few daily passes, and clouds or thick smoke can hide a fire, so a day with no detection is not proof of no fire.

Provider	NASA FIRMS (Fire Information for Resource Management System), part of NASA’s Earth Science Data and Information System (ESDIS). The detections are the MODIS active fire product (MCD14ML) of the MODIS fire team at the University of Maryland, in standard (science-quality) processing
Coverage	2003–2024 (years of the UTC date; UTC is Coordinated Universal Time, the clock at Greenwich) · Canada, the United States (with Alaska and Hawaii), and Mexico, one file per country and year
Raw unit	one row = one fire detection: one MODIS pixel (about 1 km × 1 km, larger near the edge of the satellite’s view) flagged as burning during one Terra or Aqua overpass
Clean unit	detection file: one row = one detection; province-day file: one row = one Canadian province or territory × one local solar day; country-day file: one row = one country × one local solar day. A local solar day is the calendar day by the sun at the fire (see “Cleaning steps”)
Licence and citation	NASA data are open, with no restrictions on use. FIRMS asks users to acknowledge it: “We acknowledge the use of data and/or imagery from NASA’s Fire Information for Resource Management System (FIRMS) (https://earthdata.nasa.gov/firms), part of NASA’s Land, Atmosphere Near real-time Capability for Earth observations (LANCE) (https://earthdata.nasa.gov/lance) and NASA’s Earth Science Data and Information System (ESDIS).” Data DOI: 10.5067/FIRMS/MODIS/MCD14ML. Algorithm: Giglio, L., Schroeder, W., and Justice, C.O. (2016), “The collection 6 MODIS active fire detection algorithm and fire products,” Remote Sensing of Environment 178, 31–41
Documentation	FIRMS attribute fields for MODIS · FIRMS FAQ · MODIS C6/C6.1 Active Fire Product User’s Guide (copy hosted by NASA’s LP DAAC) · FIRMS yearly country files · firms/AttributesReadme.txt (shipped)

Raw files

File (in wildfire_data/raw/)	Size	Format	One row is
firms/modis_<year>_Canada__v<date>.csv (22 files, 2003–2024)	0.98–26.47 MB each; 119.6 MB in all	comma-separated text with a header row; 15 columns	one detection that FIRMS places in Canada (1,487,842 rows)
firms/modis_<year>_United_States__v<date>.csv (22 files)	7.57–13.53 MB each; 224.1 MB in all	same	one detection that FIRMS places in the United States, Alaska and Hawaii included (2,825,073 rows)
firms/modis_<year>_Mexico__v<date>.csv (22 files)	3.25–8.62 MB each; 123.1 MB in all	same	one detection that FIRMS places in Mexico (1,556,116 rows)
firms/AttributesReadme.txt	0.003 MB	plain text	provider README: links to the attribute definitions, the coordinate system (WGS 84 latitude/longitude), and the citation and disclaimer

FIRMS posts each file as, for example, modis_2003_Canada.csv, and replaces it when it reprocesses the data. The research project added __v<date> when it downloaded the files: the date FIRMS last updated the file. It is v20231214 for 2003–2022, v20240603 for 2023, and v20250530 for 2024. Only the newest copy of each year × country is shipped (the project keeps older re-posted copies in a _superseded folder; there are none for these MODIS files). <year> is the UTC year of the overpass, and all three countries’ files for a year must be read together to get that year’s full picture. The 66 files hold 5,869,031 rows and all have the same 15 columns.

Three things to know about these files:

• The version is not the same in every year. The project’s folder for these files is called modis_c61, after MODIS Collection 6.1, but the version column says otherwise: 6.2 in the 2003–2017 files, 6.03 in the 2018–2022 files, and 61.03 in the 2023–2024 files. (The project’s source notes give the 6.03 years as 2017–2022, but no 2017 row carries 6.03.) The MODIS user guide names the underlying product files MCD14ML.YYYYMM.CCC.VV, with CCC the collection and VV the product version, so 6.03 reads as Collection 6.0, version 03, and 61.03 as Collection 6.1, version 03. FIRMS does not document the string 6.2; the research project treats it, like 6.03, as Collection 6.0 standard processing. A version without an NRT suffix is standard processing, not the quicker near-real-time (NRT) product. The changes between versions are small, but a series that crosses 2017/2018 or 2022/2023 crosses a processing change (see “Checks and cautions”).
• There is no 2025 file. FIRMS had not posted 2025 yearly MODIS files when the project downloaded them. The project’s main MODIS source is a separate FIRMS Archive Download request (request 813444, 2003–2025, made with the researcher’s NASA Earthdata login; 559 MB). It is the only copy of 2025 standard-processing data the project has, and it is not shipped. So this module stops at 31 December 2024.
• VIIRS is not shipped. The project also has fire detections from the VIIRS sensor (375 m pixels, many more detections) on the NOAA-20 satellite (2018–2025) and on Suomi NPP (2012–2024), about 2.2 GB in all. The project uses them only for checks, and they are not in this bundle.

Raw variables

All 66 CSV files have the 15 columns below, in this order. Descriptions follow the FIRMS attribute page and the MODIS user guide; value ranges and counts are over all 5,869,031 rows.

Variable	Type	Description
latitude	decimal number	Latitude of the center of the 1 km fire pixel, in degrees north (WGS 84, the coordinate system GPS uses). FIRMS notes it is not necessarily the location of the fire itself, since one or more fires can burn anywhere inside the pixel. 14.56 to 82.34 in these files
longitude	decimal number	Longitude of the pixel center, in degrees east; negative values are west of Greenwich. −176.61 to −52.89
brightness	decimal number	Brightness temperature of the fire pixel in MODIS channel 21/22 (a 4 µm infrared band), in kelvin (K). 300.0 to 510.5 K
scan	decimal number	Size of the pixel along the scan (east–west), in km. The algorithm works on 1 km pixels, but MODIS pixels get bigger toward the edge of the scan; scan and track give the actual size. 1.0 to 4.8 km
track	decimal number	Size of the pixel along the satellite’s track (north–south), in km. 1.0 to 2.0 km
acq_date	date text, YYYY-MM-DD	Date of the overpass in UTC, which runs 2.5 to 10 hours ahead of local clock time in Canada, the U.S., and Mexico
acq_time	4-digit text, HHMM	Time of the overpass in UTC, hours and minutes, e.g., 0606 = 06:06 UTC. The leading zero is written in these files
satellite	text	Terra or Aqua (5,869,031 rows: 2,838,800 Terra, 3,030,231 Aqua). The FIRMS attribute page lists the codes T and A; these files spell the names out
instrument	text	Always MODIS. Not listed in the provider’s MODIS attribute table
confidence	integer, 0–100	Detection confidence in percent, built from intermediate quantities of the detection algorithm, to help users judge the quality of each detection. The user guide sorts it into three classes: low (0–29), nominal (30–79), and high (80–100). 446,238 rows are below 30
version	decimal-looking text	Collection and processing source of the detection: 6.2 (2003–2017 files), 6.03 (2018–2022), 61.03 (2023–2024). See the note above
bright_t31	decimal number	Brightness temperature of the fire pixel in MODIS channel 31 (an 11 µm infrared band), in K. 263.5 to 400.1 K
frp	decimal number	Fire radiative power: the radiant heat output of the whole pixel, in MW. −59.2 to 16,146.4 MW; 14,875 rows are 0 and one row is negative (see “Checks and cautions”)
daynight	text	D = daytime fire, N = nighttime fire. The user guide calls it the day/night flag of the detection algorithm, which uses different tests by day and by night (4,562,070 D, 1,306,961 N)
type	integer, 0–3	Inferred hot-spot type: 0 = presumed vegetation fire (5,621,753 rows), 1 = active volcano (13,500), 2 = other static land source (224,999), 3 = offshore (8,779). Per the user guide, type 2 covers hot spots detected at the same place on 16 or more days in a calendar year (gas flares, industrial plants) and hot spots in urban areas, and type 3 covers every hot spot over water, presumably offshore gas flares

AttributesReadme.txt has no variables; it points to the documentation above.

Cleaning steps

The script makes these decisions:

1. Newest file only. It reads the newest __v<date> copy of each year × country, so a re-posted file can never be counted twice. That gives 66 files.
2. Country from the file, region from the map. FIRMS has already put each detection in a country, and the script keeps that. It then tests Canadian points only against the province outlines and U.S. points only against the state outlines from module m00 (point-in-polygon: is the point inside the polygon?). A point that falls in no polygon, on a coast or right at a border, gets the nearest polygon, and region_dist_km records how far away it was. Mexican points get MX, because the bundle has no Mexican state outlines.
3. Borders follow parallels, so the outlines are densified first. Long borders such as the 49th parallel are stored as a few corners several degrees apart. On the sphere, sf joins two corners by a great circle (the shortest path on the globe), which bows up to about 5 km toward the pole and would put points near the border in the wrong polygon (without this step, 239 Canadian detections land in the wrong province, most of them Ontario points labeled Manitoba). Adding a corner every 0.05° along the straight longitude/latitude line keeps each border on its parallel.
4. One duplicate removed. One Aqua detection in Manitoba on 29 June 2006 appears twice: same pixel, same time, same satellite, with two slightly different FRP values (1,037.4 and 1,029.6 MW). The script keeps the first row, leaving 5,869,030 detections. It counts the raw lines and checks that nothing else is dropped, without hard-coding the totals, so a refreshed FIRMS file does not break it.
5. Local solar date. Daily panels in this project count fires by the day at the fire, not by the UTC date. Local solar time is UTC plus longitude/15 hours (4 minutes per degree), the clock the sun keeps at that longitude. For example, a detection at 03:00 UTC on 2 July at longitude −120° happened at 19:00 local solar time on 1 July. This rule needs no time-zone map and treats every place the same way; it is the rule in the research project’s local_solar_date() function. For 924,679 detections (15.8%), almost all from Terra’s late-evening pass (906,304), the local date is the day before the UTC date.
6. Keep every detection; filter in the totals. The detection file keeps all types and all confidence levels, with the type and confidence columns, so you can choose your own filter. The province-day and country-day files count only presumed vegetation fires (type == 0) with nominal or high confidence (confidence >= 30), as the research project does. Types 1–3 are volcanoes, gas flares, industrial sites, urban hot spots, and offshore platforms, not wildfires. Below 30 is the provider’s low-confidence class; the user guide suggests that users who want fewer false alarms keep only nominal- and high-confidence pixels. This keeps 5,196,099 of the 5,869,030 detections (88.5%).
7. Complete grids. The two daily files cover every local day from 2003-01-01 to 2024-12-31 (8,036 days), with zeros on days without a detection.
8. What is dropped or changed. instrument is dropped (always MODIS). acq_time stays text so the leading zero survives. Detection values keep their raw precision; daily FRP sums are rounded to 0.1 MW.

List the 66 raw files and read the year, country, and provider update date from each file name.

# FIRMS publishes one CSV per year and country. The file name holds the year,
# the country, and (after "__v") the date the provider last updated the file.
# If the provider re-posts a file, two versions of the same year x country can
# sit side by side; keeping only the newest one stops us from counting the
# same fire twice.
firms_files <- tibble(
  path = list.files(file.path(raw_dir, "firms"),
                    pattern = "^modis_[0-9]{4}_.+__v[0-9]{8}\\.csv$",
                    full.names = TRUE)
) |>
  mutate(
    file = basename(path),
    file_year = as.integer(str_sub(file, 7, 10)),
    country = str_match(file, "^modis_[0-9]{4}_(.+)__v")[, 2] |>
      str_replace_all("_", " "),
    vintage = ymd(str_match(file, "__v([0-9]{8})\\.csv$")[, 2])
  ) |>
  slice_max(vintage, n = 1, by = c(file_year, country)) |>
  arrange(file_year, country)

# Expect 22 years (2003-2024) x 3 countries, one file each.
stopifnot(
  nrow(firms_files) == 66,
  !anyDuplicated(select(firms_files, file_year, country)),
  setequal(firms_files$country, c("Canada", "Mexico", "United States")),
  setequal(firms_files$file_year, 2003:2024)
)
count(firms_files, vintage)

Set the column types once, and look at one file before reading all of them.

# Spell out every column type. acq_time is text so that "0606" (06:06 UTC)
# keeps its leading zero, and version is text because it is a label
# ("6.03", "61.03"), not a number to do math with.
modis_cols <- cols(
  latitude = col_double(),
  longitude = col_double(),
  brightness = col_double(),
  scan = col_double(),
  track = col_double(),
  acq_date = col_date(),
  acq_time = col_character(),
  satellite = col_character(),
  instrument = col_character(),
  confidence = col_integer(),
  version = col_character(),
  bright_t31 = col_double(),
  frp = col_double(),
  daynight = col_character(),
  type = col_integer()
)

# Look at one file before reading 66 of them.
read_csv(firms_files$path[1], col_types = modis_cols) |>
  glimpse()

Load the province and state outlines that module m00 wrote, and add corners along their long straight edges so that borders on a parallel stay on that parallel.

# Outlines written by module m00, in longitude/latitude (WGS 84), the same
# coordinate system as the FIRMS points. We need only the two-letter code.
provinces <- read_sf(file.path(clean_dir, "canada_provinces_2021.gpkg")) |>
  select(region = prov_abbr)
states <- read_sf(file.path(clean_dir, "us_states_2020.gpkg")) |>
  select(region = state_abbr)

# 13 provinces and territories; 50 states plus DC (Alaska and Hawaii included).
stopifnot(nrow(provinces) == 13, nrow(states) == 51)

# Why add points to the borders? Many borders follow a parallel (a line of
# constant latitude), such as the 49th parallel between Canada and the U.S.
# The outlines store such a border as a few corners up to 6 degrees of
# longitude apart. For longitude/latitude data sf works on the sphere (the
# s2 library) and joins two corners by the shortest path on the globe, a
# great circle. Between corners far apart, that path bows toward the pole by
# up to about 5 km, so points near the border land in the wrong polygon.
# st_segmentize() adds a corner every 0.05 degrees along each edge. We drop
# the coordinate system for this step (st_set_crs(NA)) so the new corners
# are spaced along the straight line in longitude/latitude, then put it
# back. With corners this close, the bow is under 1 meter.
densify <- function(polygons) {
  polygons |>
    st_set_crs(NA) |>
    st_segmentize(dfMaxLength = 0.05) |>
    st_set_crs(4326)
}
provinces <- densify(provinces)
states <- densify(states)

The next four chunks only define functions. The first gives each detection its province or state code: the sf function st_intersects() finds the polygon that contains each point, and st_nearest_feature() finds the closest polygon.

# Give each detection the code of the polygon it falls in. The country comes
# from the FIRMS file name, so Canadian points are tested only against the
# province layer and U.S. points only against the state layer. (Testing both
# layers would give some border points two labels and others none; see m00.)
# A point in no polygon (on a coast, in a border lake, or near a simplified
# border) gets the nearest polygon, and region_dist_km records how far away
# that polygon is. A point inside a polygon gets region_dist_km = 0.
label_regions <- function(det, polygons) {
  pts <- st_as_sf(det, coords = c("longitude", "latitude"), crs = 4326,
                  remove = FALSE)
  # st_intersects() lists, for each point, the rows of the polygons that
  # contain it. rows[1] takes the first one and gives NA when the list is empty.
  # Because sf works on the sphere here, st_distance() returns meters along
  # the Earth's surface.
  poly_row <- map_int(st_intersects(pts, polygons), \(rows) rows[1])
  dist_km <- rep(0, nrow(det))
  outside <- is.na(poly_row)
  if (any(outside)) {
    nearest <- st_nearest_feature(pts[outside, ], polygons)
    poly_row[outside] <- nearest
    dist_km[outside] <- as.numeric(
      st_distance(pts[outside, ], polygons[nearest, ], by_element = TRUE)
    ) / 1000
  }
  det |>
    mutate(region = polygons$region[poly_row],
           region_dist_km = round(dist_km, 3))
}

The second function cleans one UTC year: it reads the three country files, removes the duplicate, labels regions, and adds the local solar date.

# Clean one year at a time: three files (about 270,000 rows) fit easily in
# memory, and the spatial step stays fast.
clean_one_year <- function(yr) {
  files_yr <- filter(firms_files, file_year == yr)
  # id = "path" adds a column with the file each row came from, so the
  # country in the file name can be attached to every row.
  det <- read_csv(files_yr$path, col_types = modis_cols, id = "path") |>
    left_join(select(files_yr, path, country), by = "path") |>
    # Same pixel, same time, same satellite = the same detection. One such
    # pair exists (Canada, 29 June 2006); keep the first row.
    distinct(latitude, longitude, acq_date, acq_time, satellite,
             .keep_all = TRUE)

  canada <- det |> filter(country == "Canada") |> label_regions(provinces)
  usa <- det |> filter(country == "United States") |> label_regions(states)
  # No sub-national layer for Mexico: every Mexican detection gets "MX".
  mexico <- det |>
    filter(country == "Mexico") |>
    mutate(region = "MX", region_dist_km = NA_real_)

  bind_rows(canada, usa, mexico) |>
    mutate(
      # Local solar time = UTC + longitude / 15 hours, i.e. 4 minutes per
      # degree. West of Greenwich the longitude is negative, so the local
      # date can be the day before the UTC date.
      utc_minutes = as.integer(str_sub(acq_time, 1, 2)) * 60 +
        as.integer(str_sub(acq_time, 3, 4)),
      local_date = acq_date + floor((utc_minutes + 4 * longitude) / 1440),
      year = as.integer(year(local_date))
    ) |>
    select(country, region, region_dist_km, latitude, longitude,
           acq_date, acq_time, local_date, year, satellite, confidence,
           frp, brightness, bright_t31, scan, track, daynight, type,
           version) |>
    arrange(country, acq_date, acq_time)
}

The third function checks one cleaned year. stopifnot() stops the script with an error if any condition is false.

# Checks for one cleaned year; stopifnot() stops the script if any is FALSE.
check_year <- function(det) {
  stopifnot(
    # One row per detection.
    !anyDuplicated(select(det, latitude, longitude, acq_date, acq_time,
                          satellite)),
    # Only Mexico has no distance (no polygon test); nothing else is missing.
    !anyNA(select(det, -region_dist_km)),
    all(is.na(det$region_dist_km) == (det$country == "Mexico")),
    all(str_detect(det$acq_time, "^[0-9]{4}$")),
    all(det$satellite %in% c("Terra", "Aqua")),
    all(det$daynight %in% c("D", "N")),
    all(det$type %in% 0:3),
    all(between(det$confidence, 0, 100)),
    # Every point lies west of Greenwich, so the local date is the UTC date
    # or the day before.
    all(as.numeric(det$local_date - det$acq_date) %in% c(-1, 0))
  )
  invisible(det)
}

The fourth function sums one year of detections into totals by region and local day.

# The daily files count presumed vegetation fires (type 0) seen with nominal
# or high confidence (confidence >= 30), as in the research project. This
# function sums one year of detections by country, region, and local day.
# n_rows counts every detection of any type or confidence; we use it only to
# check that no row was lost.
summarize_fires <- function(det) {
  det |>
    mutate(veg = type == 0 & confidence >= 30) |>
    summarize(
      n_rows = n(),
      n_detections = sum(veg),
      frp_total_mw = sum(frp[veg]),
      frp_day_mw = sum(frp[veg & daynight == "D"]),
      frp_night_mw = sum(frp[veg & daynight == "N"]),
      n_terra = sum(veg & satellite == "Terra"),
      n_aqua = sum(veg & satellite == "Aqua"),
      .by = c(country, region, local_date)
    )
}

Try the functions on 2020 before running all 22 years. In 2020, 18 Canadian and 10 U.S. detections fell outside every polygon (all within 0.77 km of one), and the largest region-days are California and Oregon on 7–9 September 2020.

# Try the functions on one year before running all 22.
try_2020 <- clean_one_year(2020) |> check_year()
try_2020 |>
  summarize(detections = n(), outside_polygon = sum(region_dist_km > 0),
            max_dist_km = max(region_dist_km), .by = country)
summarize_fires(try_2020) |>
  arrange(desc(frp_total_mw)) |>
  head(5)

Run all 22 years. Each year is written to the detection file as soon as it is cleaned, so memory stays low. The chunk first counts the raw data lines, then checks that only exact duplicates were removed.

# All 22 years hold about 5.9 million detections, close to 1 GB in R. To keep
# memory low, each year is cleaned, checked, appended to one compressed file
# on disk, and summed by region and day; only the small daily sums stay in
# memory. Takes about 5 minutes.
# File grain: one row per detection (key: latitude, longitude, acq_date,
# acq_time, satellite), all types, all confidence levels, all three
# countries. Each raw file holds one UTC year, so a key cannot repeat across
# years. As plain text the file would be about 600 MB, so it is gzipped;
# read_csv() opens .csv.gz files directly.
det_file <- file.path(clean_dir, "firms_modis_detections_2003_2024.csv.gz")

# Count the data lines in the raw files (each file has one header line), so
# we can check below that cleaning drops only exact duplicates.
raw_rows <- map_int(firms_files$path, \(f) length(read_lines(f)) - 1L) |>
  sum()

region_day_pieces <- map(2003:2024, \(yr) {
  det_yr <- clean_one_year(yr) |> check_year()
  stopifnot(all(year(det_yr$acq_date) == yr))
  # 2003 creates the file with a header line; append = TRUE adds each later
  # year below it, without a header.
  write_csv(det_yr, det_file, na = "", append = yr > 2003)
  summarize_fires(det_yr)
}) |>
  list_rbind()

# A night detection at 03:00 UTC on 1 January has a local date of
# 31 December, so two UTC years can both add to the same local day. Summing
# again by region and day joins those two pieces.
region_day_all <- region_day_pieces |>
  summarize(across(n_rows:n_aqua, sum), .by = c(country, region, local_date))

# Every raw row is kept except exact duplicates. With the files shipped in
# 2026 that is 5,869,031 raw rows and 1 duplicate removed (5,869,030 kept).
# A refreshed FIRMS file changes these numbers, so we do not hard-code them.
n_removed <- raw_rows - sum(region_day_all$n_rows)
stopifnot(between(n_removed, 0, 100),
          !anyDuplicated(select(region_day_all, country, region, local_date)))
c(raw_rows = raw_rows, duplicates_removed = n_removed)
region_day_all |>
  summarize(all_rows = sum(n_rows), veg_fires = sum(n_detections),
            .by = country)

Keep the local days of 2003–2024. Only two vegetation-fire detections fall on 31 December 2002 (one Canadian, one U.S.).

# Keep local days 2003-01-01 to 2024-12-31. The only days outside are
# 31 Dec 2002 (detections from the first UTC hours of 1 Jan 2003).
first_day <- ymd("2003-01-01")
last_day <- ymd("2024-12-31")
region_day_all |>
  filter(!between(local_date, first_day, last_day)) |>
  summarize(veg_fires = sum(n_detections), .by = c(country, local_date))
region_day <- filter(region_day_all, between(local_date, first_day, last_day))
all_days <- seq(first_day, last_day, by = "day")

Build the complete province × day grid, check it, and write it.

# Complete grid: every province x every local day. A day with no detection
# has no row in region_day, so after the join it gets zeros: no fire seen.
province_day <- expand_grid(prov_abbr = provinces$region,
                            local_date = all_days) |>
  left_join(region_day |>
              filter(country == "Canada") |>
              select(prov_abbr = region, local_date, n_detections:n_aqua),
            by = c("prov_abbr", "local_date")) |>
  mutate(
    across(c(n_detections, n_terra, n_aqua), \(x) replace_na(x, 0L)),
    across(starts_with("frp_"), \(x) round(replace_na(x, 0), 1))
  ) |>
  arrange(prov_abbr, local_date)

# Grain: one row per province x local day; 13 x 8,036 days.
stopifnot(
  nrow(province_day) == 13 * 8036,
  !anyDuplicated(select(province_day, prov_abbr, local_date)),
  !anyNA(province_day),
  sum(province_day$n_detections) ==
    sum(region_day$n_detections[region_day$country == "Canada"]),
  all(province_day$n_terra + province_day$n_aqua ==
        province_day$n_detections),
  all(abs(province_day$frp_day_mw + province_day$frp_night_mw -
            province_day$frp_total_mw) < 0.15),
  all(province_day$frp_total_mw >= 0)
)
write_csv(province_day,
          file.path(clean_dir, "firms_canada_province_day_2003_2024.csv"),
          na = "")

Sum by country and local day, and split the U.S. into the contiguous states and Alaska.

# Country totals by local day (only days with at least one detection).
country_day <- region_day |>
  summarize(n_detections = sum(n_detections),
            frp_total_mw = sum(frp_total_mw),
            .by = c(country, local_date))

# U.S. split: contiguous U.S. (48 states + DC) and Alaska. The rest of the
# U.S. total is Hawaii. Smoke from Alaska rarely reaches the lower 48, so
# users often want the contiguous part on its own.
us_split <- region_day |>
  filter(country == "United States") |>
  mutate(contig = !region %in% c("AK", "HI")) |>
  summarize(
    n_contig = sum(n_detections[contig]),
    frp_contig_mw = sum(frp_total_mw[contig]),
    n_alaska = sum(n_detections[region == "AK"]),
    frp_alaska_mw = sum(frp_total_mw[region == "AK"]),
    .by = local_date
  ) |>
  mutate(country = "United States")

Build the complete country × day grid, check it, and write it.

# Complete grid: every country x every local day. A missing count means no
# fire seen (0). The split columns do not apply to Canada and Mexico (NA).
country_day_full <- expand_grid(
  country = c("Canada", "United States", "Mexico"),
  local_date = all_days
) |>
  left_join(country_day, by = c("country", "local_date")) |>
  left_join(us_split, by = c("country", "local_date")) |>
  mutate(
    n_detections = replace_na(n_detections, 0L),
    frp_total_mw = round(replace_na(frp_total_mw, 0), 1),
    across(c(n_contig, n_alaska),
           \(x) if_else(country == "United States", replace_na(x, 0L), NA)),
    across(c(frp_contig_mw, frp_alaska_mw),
           \(x) if_else(country == "United States",
                        round(replace_na(x, 0), 1), NA))
  )

# Grain: one row per country x local day; 3 x 8,036 days.
us_rows <- filter(country_day_full, country == "United States")
stopifnot(
  nrow(country_day_full) == 3 * 8036,
  !anyDuplicated(select(country_day_full, country, local_date)),
  !anyNA(select(country_day_full, country, local_date, n_detections,
                frp_total_mw)),
  !anyNA(us_rows),
  sum(country_day_full$n_detections) == sum(region_day$n_detections),
  all(us_rows$n_contig + us_rows$n_alaska <= us_rows$n_detections)
)
write_csv(country_day_full,
          file.path(clean_dir, "firms_country_day_2003_2024.csv"),
          na = "")

A first look: Canadian vegetation-fire detections and FRP by year. 2023 has 280,489 detections, more than twice 2024 and more than three times any earlier year.

# A first look: Canadian fire activity by year. 2023 stands out.
province_day |>
  mutate(year = year(local_date)) |>
  summarize(n_detections = sum(n_detections),
            frp_total_gw = round(sum(frp_total_mw) / 1000, 1),
            .by = year) |>
  print(n = 22)

How to read the detection file in your own work: name only the columns you need, and read the file in pieces so that memory stays low. The checks confirm that the file has every Canadian detection and that the vegetation fires among them add up to the province-day file.

# How to read the detection file later. All 19 columns of 5.9 million rows
# need more than 2 GB of memory while reading. Two habits keep it small:
# cols_only() reads just the columns you name, with their types, and
# read_csv_chunked() reads 250,000 rows at a time and keeps only what the
# callback function returns (here, the Canadian rows). If you add acq_time
# or version, read them with col_character(): acq_time must keep its 4
# characters, and version is a label ("6.2", "61.03"), not a number.
fires_ca <- read_csv_chunked(
  det_file, chunk_size = 250000,
  col_types = cols_only(country = col_character(), region = col_character(),
                        local_date = col_date(), confidence = col_integer(),
                        frp = col_double(), type = col_integer()),
  callback = DataFrameCallback$new(\(x, pos) filter(x, country == "Canada"))
)

# Every Canadian detection is there, and the vegetation fires among them
# add up to the province-day file.
veg_ca <- filter(fires_ca, type == 0, confidence >= 30,
                 between(local_date, first_day, last_day))
stopifnot(
  nrow(fires_ca) ==
    sum(region_day_all$n_rows[region_day_all$country == "Canada"]),
  nrow(veg_ca) == sum(province_day$n_detections),
  abs(sum(veg_ca$frp) - sum(province_day$frp_total_mw)) < 1
)
count(veg_ca, region, sort = TRUE)

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
firms_modis_detections_2003_2024.csv.gz	5,869,030	one MODIS detection: all types, all confidence levels, all three countries; sorted by country, acq_date, acq_time	112.1 MB (614.3 MB uncompressed)
firms_canada_province_day_2003_2024.csv	104,468	one Canadian province or territory × one local solar day (13 × 8,036 days); sorted by prov_abbr, local_date	3.0 MB
firms_country_day_2003_2024.csv	24,108	one country × one local solar day (3 × 8,036 days); sorted by country (Canada, United States, Mexico), then local_date	0.9 MB

Clean variables

firms_modis_detections_2003_2024.csv.gz: one row per detection; the key is latitude, longitude, acq_date, acq_time, satellite. Types are those you get with col_types = cols(acq_time = col_character(), version = col_character(), year = col_integer(), confidence = col_integer(), type = col_integer()). Left to guess, read_csv() reads year, confidence, type, and version as dbl, which turns the version labels into numbers. Reading all 19 columns at once takes about 0.8 GB in R and more than 2 GB while reading; chunk m01-read-back shows how to read only what you need.

Variable	Type	Units	Description
country	chr	—	Country of the FIRMS file the detection came from: Canada, United States, or Mexico
region	chr	—	Two-letter code of the province or territory (Canada; 13 codes, e.g., BC, QC) or state (U.S.; 50 states and DC, with AK and HI) that contains the point, or the nearest one when no polygon contains it; MX for every Mexican detection. Codes join to prov_abbr in canada_provinces_2021.csv and state_abbr in us_states_2020.csv
region_dist_km	dbl	km	Distance from the point to the polygon in region: 0 inside a polygon; above 0 for points given the nearest polygon (639 Canadian and 204 U.S. detections; at most 1.91 km). Empty (NA) for Mexico: not applicable, because no polygon test is made
latitude	dbl	degrees north	Center of the fire pixel (raw)
longitude	dbl	degrees east	Center of the fire pixel (raw); negative = west
acq_date	date	—	UTC date of the overpass (raw); 2003-01-01 to 2024-12-31
acq_time	chr	HHMM	UTC time of the overpass, always 4 characters, e.g., 0606
local_date	date	—	Local solar date at the fire: acq_date and acq_time plus longitude/15 hours; 2002-12-31 to 2024-12-31. Equal to acq_date or one day earlier
year	int	—	Year of local_date; 2002 for the 12 detections on 31 December 2002
satellite	chr	—	Terra or Aqua
confidence	int	%	Detection confidence, 0–100 (raw): low 0–29, nominal 30–79, high 80–100
frp	dbl	MW	Fire radiative power of the pixel (raw); −59.2 to 16,146.4
brightness	dbl	K	Channel 21/22 brightness temperature (raw)
bright_t31	dbl	K	Channel 31 brightness temperature (raw)
scan	dbl	km	Pixel size along the scan, east–west (raw)
track	dbl	km	Pixel size along the track, north–south (raw)
daynight	chr	—	D = daytime fire, N = nighttime fire (raw)
type	int	—	Hot-spot type (raw): 0 presumed vegetation fire, 1 active volcano, 2 other static land source, 3 offshore
version	chr	—	Collection and processing label (raw): 6.2, 6.03, or 61.03

The two daily files count only presumed vegetation fires (type == 0) with confidence >= 30, by local solar day. read_csv() guesses every type correctly; counts come in as dbl.

firms_canada_province_day_2003_2024.csv:

Variable	Type	Units	Description
prov_abbr	chr	—	Two-letter province or territory code (13 codes; joins to canada_provinces_2021.csv)
local_date	date	—	Local solar day, 2003-01-01 to 2024-12-31
n_detections	dbl	count	Number of vegetation-fire detections. 0 means no detection, not necessarily no fire
frp_total_mw	dbl	MW	Sum of frp over those detections, rounded to 0.1. A sum over separate snapshots, not an energy total (see below)
frp_day_mw	dbl	MW	Part of frp_total_mw from daytime detections (daynight == "D")
frp_night_mw	dbl	MW	Part of frp_total_mw from nighttime detections (daynight == "N")
n_terra	dbl	count	Detections by Terra
n_aqua	dbl	count	Detections by Aqua; n_terra + n_aqua == n_detections

firms_country_day_2003_2024.csv:

Variable	Type	Units	Description
country	chr	—	Canada, United States, or Mexico (the FIRMS file’s country)
local_date	date	—	Local solar day, 2003-01-01 to 2024-12-31
n_detections	dbl	count	Number of vegetation-fire detections; 0 = no detection
frp_total_mw	dbl	MW	Sum of frp over those detections, rounded to 0.1
n_contig	dbl	count	U.S. rows: detections in the 48 contiguous states and DC. Empty (NA) for Canada and Mexico: not applicable
frp_contig_mw	dbl	MW	U.S. rows: FRP in the contiguous states and DC; NA for Canada and Mexico
n_alaska	dbl	count	U.S. rows: detections in Alaska; NA for Canada and Mexico
frp_alaska_mw	dbl	MW	U.S. rows: FRP in Alaska; NA for Canada and Mexico. Hawaii is the rest: n_detections − n_contig − n_alaska

Checks and cautions

• Agreement with the research project (Canada). R/check_m01.R compares the Canadian totals with the project’s main MODIS file, which comes from a different FIRMS delivery (archive request 813444) and places detections in countries and provinces with the project’s own outlines. By local year, the counts here differ from the project’s by −0.01% to +0.66%, and FRP by −0.01% to +1.23%. Over 2003–2024 this module has 1,279,815 Canadian vegetation-fire detections against 1,277,685 in the project (+0.17%), and 149.30 million MW of summed FRP against 149.18 million (+0.08%). The yearly table is in logs/check_m01_canada_by_year.csv.
• Why the two differ. From 2018 on, the country files and the archive carry the same version, and the detections match one for one: all 624,552 Canadian vegetation-fire detections of 2018–2024 here are in the project file, and 624,331 of them (99.96%) get the same province. The project puts 160 of the rest in the United States (153 of them are points this module snapped to the nearest province) and 61 in a different province. Before 2018 the country files (version 6.2) and the archive (6.03) are different processing runs, and the country files have 0.1–0.7% more Canadian detections per year. By province over 2003–2024 the gap is under 0.3% for the seven provinces and territories with the most fires; Ontario is +1.65% (45,687 against 44,947), and each of the small Atlantic provinces is within 1%.
• Agreement with the project’s cleaning of these same files. The project also cleaned the same 66 country files as a cross-check (firms_archive_vs_country_check.csv). Against it, the country × local-year counts agree within 0.15% for Canada, 0.08% for the U.S. without Hawaii (the project drops Hawaii), and 0.04% for Mexico; FRP within 1.23%, 0.51%, and 0.05%. The small gaps come from the country rule: the project moves each detection to the country whose polygon contains it, and this module keeps the FIRMS country.
• Detections across the border. FIRMS’s country borders are not exactly the Census and Statistics Canada outlines. 639 detections from the Canadian files and 204 from the U.S. files lie outside every polygon of their own country (half of them within about 0.35 km, all within 1.91 km) and got the nearest province or state. 549 of the 639 Canadian ones lie inside a U.S. state polygon; 494 of those are vegetation fires, with 84,717 MW over 22 years. On the 49th-parallel stretch of the border, 556 of the 568 snapped Canadian points lie south of 49.0°N, so they really are on the U.S. side; the other 90 snapped Canadian points lie in no polygon of either country (on coasts, in lakes, or in slivers between the two outlines). The largest group is British Columbia in 2022: 43 vegetation-fire detections with 41,662 MW, all just south of 49.0°N. That is exactly why Canada’s 2022 FRP is 1.23% above the project’s, which places those points in the United States. Only 1 of the 204 snapped U.S. points lies inside a province. In the other direction, 12 detections that the project places in Yukon are in the FIRMS U.S. file, labeled AK. If you want a strict border, drop rows with region_dist_km > 0: for Canada that removes 549 points that lie in the U.S. and 90 points that are in neither country’s outline, a negligible share of any province’s total.
• What a zero or a missing value means. In the two daily files, 0 means “no vegetation-fire detection that day”, not “no fire”: clouds, thick smoke, and the hours between passes hide fires. 25,452 of the 104,468 province-days have at least one detection. In the detection file, the only missing values are region_dist_km for Mexico (not applicable). In the country-day file, the four split columns are missing for Canada and Mexico (not applicable).
• Days when a satellite was out. On 52 UTC days, one satellite has no detection at all in the three countries’ files: Terra on 35 days and Aqua on 17, never both. The long outages are Terra 17–23 December 2003, Terra 19–27 February 2016, Aqua 1–16 April 2022, and Terra 11–23 October 2022. On those days counts and FRP are roughly half of what both satellites would have seen. These are the same 50 days on which the research project finds a satellite absent in its archive file (plus 14 March 2003 and 1 January 2023, when the project’s file has a few detections). The project fills the missing satellite’s share; this module does not.
• Counts of snapshots, not of fires. Each detection is one pixel at one moment. One large fire produces many detections on every pass, and pixels near the edge of the satellite’s view are larger (scan up to 4.8 km), so counts are not numbers of fires. Summing FRP over a day’s detections adds up four or more snapshots of power (MW), not the energy released over the day; use frp_total_mw as an index of fire activity. Night detections carry 24.5% of Canada’s FRP over 2003–2024 (from 10.7% in 2007 to 35.9% in 2019), and Aqua makes 47.4% of Canada’s detections.
• Processing changes. The version changes between the 2017 and 2018 files (6.2 to 6.03) and again for 2023 (Collection 6.0 to 6.1). For 2003–2017 the country files have 1.9–3.9% more U.S. and 0.9–2.3% more Mexican detections per year than the archive’s 6.03 processing of the same years, while for 2018–2024 the two agree within 0.05%. So part of any drop in the U.S. and Mexican counts from 2017 to 2018 can come from processing, not from fewer fires. For Canada the same gap is at most 0.7%.
• The edges of the window. The detection file has 12 detections on local 31 December 2002 (year = 2002), from the first UTC hours of 1 January 2003; the daily files leave them out. The last day, 31 December 2024, lacks the detections from the first UTC hours of 1 January 2025, which are in no shipped file. The project’s archive has 4 such detections, 2 in the U.S. and 2 in Mexico, none in Canada.
• Odd values. One detection has negative FRP: −59.2 MW (Idaho, 10 August 2013, confidence 0, so it is not in the daily totals). 14,875 detections have FRP 0; 13,840 of them are vegetation fires with confidence of at least 30, so they add to n_detections but not to FRP.
• The biggest days. Canada’s 2023 season has 280,489 vegetation-fire detections, more than twice 2024 (127,920) and more than three times any earlier year; 2020 is the quietest year (11,035). The largest province-days by FRP are Alberta and British Columbia on 22 September 2023 (4,642 and 4,383 detections, 1.64 and 1.53 million MW). About 5,700 of these detections come from two Terra night passes over British Columbia and Alberta (04:15 and 05:53 UTC on 23 September), so check the raw rows before you build a story on that day. The research project’s archive has the same detections.
• Running time. The script takes about 4 minutes and needs up to about 1.6 GB of memory, most of it in the m01-clean-all-years and m01-read-back chunks.

Canadian burned area (CWFIS NBAC)

The National Burned Area Composite (NBAC) maps the areas burned by wildland fires in Canada in each year from 1972 to 2025. Each fire is a polygon (its outline on the map) with an area in hectares (1 ha = 10,000 m² = 0.01 km²) and a few facts about it: start and end dates, cause, and whether it was a prescribed burn. The Canadian Forest Service of Natural Resources Canada builds it for the Canadian Wildland Fire Information System (CWFIS). For each fire, a set of rules picks the best available outline: either one drawn by the provincial, territorial, or Parks Canada fire agency, or one that the Canadian Forest Service mapped from satellite images.

For this project, NBAC tells you how much of Canada burned, in which province or territory, and in which year. You can use it to rank fire seasons (2023 is by far the largest on record) and to line up big Canadian fire years with smoke in U.S. counties. It cannot tell you on which days a fire grew or how much smoke it made: each record has at most a start date and an end date, and many early records have no date at all. For daily fire activity, use the satellite fire detections (FIRMS).

Provider	Canadian Forest Service, Natural Resources Canada, Canadian Wildland Fire Information System (CWFIS). Release 2026-05-13 (layer NBAC_merged_1972_to_2025_20260513), downloaded on 2026-09-27
Coverage	fire years 1972–2025 · all of Canada (10 provinces and 3 territories)
Raw unit	one row (polygon) = one fire inside one jurisdiction: a fire that crosses a provincial, territorial, or national park boundary is cut into one record per jurisdiction
Clean unit	one row = an NBAC record (fires file); a province or territory × fire year (province-year file)
Licence and citation	Open Government Licence – Canada (licence.txt). Cite: Canadian Forest Service. Canadian Wildland Fire Information System (CWFIS), Natural Resources Canada, Canadian Forest Service, Northern Forestry Centre, Edmonton, Alberta. https://cwfis.cfs.nrcan.gc.ca, and name the release (NBAC 20260513)
Documentation	NBAC_1972to2025_20260513_shp_metadata.pdf in the raw folder; CWFIS NBAC download folder; Skakun et al. (2021), area adjustment; MAFiMS burned area mapping

Raw files

File (in wildfire_data/raw/)	Size	Format	One row is
nbac/NBAC_1972to2025_20260513_shp.zip	1,198.8 MB	zip archive holding one Esri shapefile (9 files, 1.9 GB unzipped; listed below)	a burned-area polygon: one fire inside one jurisdiction (52,610 records)
nbac/NBAC_1972to2025_20260513_shp_metadata.pdf	3.3 MB	PDF, 14 pages (a printed web page; the text is an image and cannot be searched)	the provider’s metadata: field definitions, code lists, and version history (no data)
nbac/licence.txt	0.001 MB	plain text, English and French	the licence and the citation the provider asks for (no data)

A shapefile is not one file but several files that share one name and differ in their extension. The zip holds these nine:

File inside the zip	Size (unzipped)	Format	What it holds
NBAC_merged_1972_to_2025_20260513.shp	1,789.0 MB	Esri shapefile geometry	the fire outlines (polygons), one shape per record
NBAC_merged_1972_to_2025_20260513.shx	0.4 MB	Esri shape index	where each record’s shape starts in the .shp file
NBAC_merged_1972_to_2025_20260513.dbf	133.3 MB	dBASE table	the attribute table: one row per polygon, 19 columns (described below)
NBAC_merged_1972_to_2025_20260513.prj	< 0.01 MB	text	the map projection: Canada Lambert Conformal Conic on the NAD83 datum, in meters
NBAC_merged_1972_to_2025_20260513.cpg	< 0.01 MB	text	the text encoding of the .dbf: UTF-8
NBAC_merged_1972_to_2025_20260513.sbn, .sbx	0.5 MB, 0.02 MB	Esri spatial index	speeds up map searches; not needed to read the data
NBAC_merged_1972_to_2025_20260513.shp.xml	0.04 MB	ArcGIS metadata, XML	the layer’s metadata in machine-readable form
NBAC_merged_1972_to_2025_20260513.pdf	3.3 MB	PDF	the same file as the metadata PDF in the raw folder (identical bytes)

Raw variables

The metadata PDF and licence.txt hold no data. This table covers the 19 columns of the attribute table (.dbf) inside the zip. The descriptions follow the metadata PDF; counts are from this release. The metadata also lists two fields that are not columns of the .dbf: FID, an internal feature number that ArcGIS generates, and Shape, the polygon itself (stored in the .shp file). foreign::read.dbf() returns dBASE F fields as dbl, C fields as chr, and D fields as Date.

Variable	Type	Description
YEAR	number (dBASE F, double)	The fire year, 1972–2025
NFIREID	number (dBASE F, double)	Fire number. Unique for each fire within one year (not across years). It is the common ID that links the pieces of a fire that crosses a provincial, territorial, or national park boundary
BASRC	text (dBASE C)	Burned area source: the product the outline comes from. MAFiMS = Multi-Acquisition Fire Mapping System, a burned area product of the Canadian Forest Service, doi:10.1071/WF19201 (13,909 records); C2C = Composite to Change, a burned area product of the Canadian Forest Service, doi:10.1016/j.rse.2017.03.035 (6,511); TFMS = Tundra Fire Mapping System, a burned area product of the Canadian Forest Service, doi:10.3390/rs16020230 (205); Agency = an outline from a provincial, territorial, or Parks Canada fire management agency (31,985)
FIREMAPS	text (dBASE C)	Data source or platform used to identify the burn. Undefined = not defined; Ground survey = ground or field survey; Aerial survey = from a helicopter or fixed-wing aircraft; Aerial photography = digital or film aerial photographs; Satellite hotspots = fire detections from satellites (MODIS, AVHRR, VIIRS); Satellite undefined = satellite, sensor not defined; MODIS = MODIS satellite images; Landsat = Landsat 5, 7, 8, or 9 images (30 m pixels); Landsat MSS = Landsat Multi-Spectral Scanner images (60 m pixels); Sentinel-2 = Sentinel-2 satellite images; Other sources = a defined source outside these categories (e.g., FireMapper); Drone (UAV) = unmanned aerial vehicle
FIREMAPM	text (dBASE C)	Method used to draw the outline. Undefined = not defined; Manually delineated = sketched or drawn by a person on a map or screen; Direct from GPS = traced with a GPS unit; Processed imagery = drawn by automated methods from satellite or aerial images; Buffered points = a circle drawn around the fire’s location (no real outline)
FIRECAUS	text (dBASE C)	Cause of ignition recorded by the agency. Natural = most often lightning (26,443 records); Human = for example industrial or non-industrial activity, prescribed burns, and arson (13,208); Undetermined = not known (12,959)
HS_SDATE	date (dBASE D)	Date of the first satellite hotspot inside the fire’s outline. A hotspot is a spot on the ground that a satellite sensor detects as burning. Empty if no hotspot was detected. AVHRR hotspots are used for 1989–2000 and MODIS hotspots from 2001; there are none before 1989. Filled for 16,123 records
HS_EDATE	date (dBASE D)	Date of the last satellite hotspot inside the fire’s outline. Same rules as HS_SDATE
AG_SDATE	date (dBASE D)	Fire start date reported by the agency. It may also be the date the agency recorded or detected the fire. For a fire that crosses a border, NBAC uses the earliest of the agencies’ dates. Empty if the agency gave no date (12,463 records)
AG_EDATE	date (dBASE D)	Fire end date reported by the agency. For a fire that crosses a border, NBAC uses the latest date. Empty if the agency gave no date (33,638 records)
CAPDATE	date (dBASE D)	Date the source data were collected: the GPS track, the air photo, or the satellite image. Empty if not provided (27,892 records). It is not a fire date: 4,169 records have an image from the year after the fire
POLY_HA	number (dBASE F, double)	Area of the polygon in hectares, calculated in the Canada Albers Equal Area Conic projection
ADJ_HA	number (dBASE F, double)	Adjusted area burned, in hectares: the official burned area. It equals POLY_HA except where ADJ_FLAG is true; there a model reduces the area of an agency outline drawn by conventional methods, which tends to include unburned patches. The models are described in Skakun et al. (2021, Environmental Research Letters 16: 064014)
ADJ_FLAG	text (dBASE C)	true when an area adjustment model was applied (2,579 records, all from agencies); empty otherwise
ADMIN_NAME	text (dBASE C)	Where the outline is mapped: a province or territory (AB Alberta, BC British Columbia, MB Manitoba, NB New Brunswick, NL Newfoundland and Labrador, NS Nova Scotia, NT Northwest Territories, NU Nunavut, ON Ontario, PE Prince Edward Island, QC Quebec, SK Saskatchewan, YT Yukon), PC = Parks Canada (2,315 records), or DND = Department of National Defence (102 records)
ADMIN_DIV	text (dBASE C)	For PC and DND records only: a 2-letter code for the national park, national historic site, or military base (Canadian Forces Base, CFB) where the outline is mapped. Empty for province and territory records. 33 codes appear in the data (26 Parks Canada, 7 Defence); the metadata lists them all. Two traps: GB means Georgian Bay Islands National Park for PC but CFB Goose Bay for DND, and NA (Nahanni National Park Reserve) is a real code, not a missing value
PRESCRIBED	text (dBASE C)	true when the agency reports a prescribed fire (a fire set on purpose, for example to reduce fuel or restore habitat); empty otherwise. 411 records
VERSION	text (dBASE C)	Version of the annual dataset: 20260513 on every record
GID	text (dBASE C)	Global identifier: the fire year and NFIREID joined by _, e.g., 2023_346. Unique to one fire across all years; the pieces of one fire share it

Cleaning steps

The script R/m10_nbac.R makes these decisions:

1. Read the attribute table, not the map. The polygons take 1,789 MB once unzipped, but fire counts and areas need only the 133 MB .dbf table. The script unzips that one file to R’s temporary folder and reads it with foreign::read.dbf(). The only polygons it reads are the 770 records of Wood Buffalo National Park (step 3).
2. Keep every record and keep it once. 450 of the 52,152 fires cross a provincial, territorial, or national park boundary, so they have two or three records that share one fire_id (908 records in all). These are pieces of one fire, not duplicates: each lies in a different jurisdiction, and their areas add up. The clean file keeps one row per piece, so each piece keeps its own province, and n_records tells you how many pieces the fire has. No fire has two records in the same jurisdiction, so nothing is dropped.
3. Give every record a province or territory. For 50,193 records, ADMIN_NAME is already a province or territory. 2,417 records belong to Parks Canada or National Defence; a lookup table gives the province of each park or base. Wood Buffalo National Park lies in both Alberta and the Northwest Territories, so for its 770 records the script takes a point inside each fire’s outline and finds the province or territory that contains it. prov_source records which of the three rules was used.
4. Use the official area. area_ha is the adjusted area (ADJ_HA), the area that NBAC reports. The polygon area (POLY_HA) is kept as mapped_area_ha. Both are rounded to 0.01 ha.
5. One start date, with its source. start_date is the agency start date when it falls in the fire year; otherwise it is the date of the first satellite hotspot; otherwise it is empty. Agencies know when a fire was reported, but hotspots exist only from 1989 on. All four raw dates are kept as well.
6. A complete province × year panel. The panel has all 13 provinces and territories × 54 years. A province-year with no fire is a row of zeros, not a missing row. Prescribed burns are left out of n_fires and burned_area_ha and reported in their own columns. The area of fires that started in May–September gets its own column, and so does the area of fires with no start date, so you can see how much is unknown.

The first chunk names the zip file and the layer inside it, and reads the 13 provinces and territories written by module m00.

dir.create(clean_dir, recursive = TRUE, showWarnings = FALSE)

nbac_zip <- file.path(raw_dir, "nbac", "NBAC_1972to2025_20260513_shp.zip")
# Every file inside the zip has this name, with a different extension.
nbac_layer <- "NBAC_merged_1972_to_2025_20260513"
years <- 1972:2025

# The 13 provinces and territories, written by module m00.
provinces <- read_csv(file.path(clean_dir, "canada_provinces_2021.csv"),
                      col_types = cols_only(prov_abbr = col_character(),
                                            prov_name = col_character()))
stopifnot(nrow(provinces) == 13, !anyDuplicated(provinces$prov_abbr))

List the files inside the zip without unzipping it. The outlines (.shp) take 1,789 MB; the attribute table (.dbf) takes 133 MB.

# A shapefile is several files that share one name. The .shp file holds the
# polygons (1,789 MB unzipped); the .dbf file holds the attribute table, one
# row per polygon (133 MB). For fire counts and areas we need only the .dbf.
unzip(nbac_zip, list = TRUE) |>
  mutate(MB = round(Length / 1024^2, 1)) |>
  select(Name, MB)

Unzip only the .dbf file and read it as a table. The check confirms the 52,610 records and 19 columns that the metadata promises; glimpse() shows the columns and their types.

# Unzip only the .dbf into R's temporary folder, then read it as a table.
# as.is = TRUE keeps text as text instead of turning it into factors.
dbf_file <- unzip(nbac_zip, files = paste0(nbac_layer, ".dbf"), exdir = tempdir())
nbac_raw <- read.dbf(dbf_file, as.is = TRUE) |>
  as_tibble()

# The provider's metadata reports 52,610 features (polygons) and 19 attributes.
stopifnot(nrow(nbac_raw) == 52610, ncol(nbac_raw) == 19)
glimpse(nbac_raw)

Most fires have one record, but some have two or three, one for each jurisdiction they burned in: 51,702 fires have one record, 442 have two, and 8 have three. The checks confirm that GID is always the year plus NFIREID, and that no fire appears twice in the same jurisdiction.

# GID = fire year + "_" + NFIREID identifies one fire. When a fire crosses a
# provincial, territorial, or national park boundary, NBAC cuts it into one
# record per jurisdiction, and all pieces share the GID. So a GID can have
# several rows. No GID has two rows for the same jurisdiction.
nbac_raw |>
  count(GID, name = "records") |>
  count(records, name = "fires")
stopifnot(!anyDuplicated(select(nbac_raw, GID, ADMIN_NAME, ADMIN_DIV)),
          all(nbac_raw$GID == paste0(nbac_raw$YEAR, "_", nbac_raw$NFIREID)))

ADMIN_NAME is not always a province. This table lists the 33 parks and military bases that appear in the data and their number of records. Wood Buffalo (WB, 770 records), Banff (BA, 290), and Jasper (JA, 239) have the most.

# ADMIN_NAME is the jurisdiction where a record is mapped: a province or
# territory code, or PC (Parks Canada) or DND (Department of National
# Defence). For PC and DND rows, ADMIN_DIV names the park or military base.
# Careful: Nahanni National Park Reserve has the code "NA". read.dbf() keeps
# it as the text "NA", which is not the same as a missing value (<NA>).
nbac_raw |>
  filter(ADMIN_NAME %in% c("PC", "DND")) |>
  count(ADMIN_NAME, ADMIN_DIV)

A small lookup table, typed in from the metadata PDF, gives the province or territory of each park and base. The check stops the script if a park or base in the data is missing from the lookup.

# Province or territory of each park and base that appears in the data. The
# names are shortened from the metadata PDF, which also lists parks that never
# appear in the data. The metadata uses the code "GB" twice,
# for Georgian Bay Islands National Park (PC) and for CFB Goose Bay (DND),
# so we join on both columns. Wood Buffalo National Park (WB) lies in both
# Alberta and the Northwest Territories, so its province is left empty here
# and found from the polygons in the next chunk.
park_lookup <- tribble(
  ~ADMIN_NAME, ~ADMIN_DIV, ~park_or_base,                                           ~park_prov,
  "DND",       "DD",       "CFB Dundurn",                                           "SK",
  "DND",       "GB",       "CFB Goose Bay",                                         "NL",
  "DND",       "GT",       "CFB Gagetown",                                          "NB",
  "DND",       "PW",       "CFB Petawawa",                                          "ON",
  "DND",       "SF",       "CFB Suffield",                                          "AB",
  "DND",       "VA",       "CFB Valcartier",                                        "QC",
  "DND",       "WW",       "CFB Wainwright",                                        "AB",
  "PC",        "BA",       "Banff National Park",                                   "AB",
  "PC",        "CB",       "Cape Breton Highlands National Park",                   "NS",
  "PC",        "EI",       "Elk Island National Park",                              "AB",
  "PC",        "GH",       "Gwaii Haanas National Park Reserve",                    "BC",
  "PC",        "GL",       "Glacier National Park",                                 "BC",
  "PC",        "GR",       "Grasslands National Park",                              "SK",
  "PC",        "IV",       "Ivvavik National Park",                                 "YT",
  "PC",        "JA",       "Jasper National Park",                                  "AB",
  "PC",        "KE",       "Kejimkujik National Park",                              "NS",
  "PC",        "KG",       "Kouchibouguac National Park",                           "NB",
  "PC",        "KL",       "Kluane National Park Reserve",                          "YT",
  "PC",        "KO",       "Kootenay National Park",                                "BC",
  "PC",        "LM",       "La Mauricie National Park",                             "QC",
  "PC",        "MI",       "Mingan Archipelago National Park Reserve",              "QC",
  "PC",        "NA",       "Nahanni National Park Reserve",                         "NT",
  "PC",        "PA",       "Prince Albert National Park",                           "SK",
  "PC",        "PP",       "Point Pelee National Park",                             "ON",
  "PC",        "PU",       "Pukaskwa National Park",                                "ON",
  "PC",        "RE",       "Mount Revelstoke National Park",                        "BC",
  "PC",        "RM",       "Riding Mountain National Park",                         "MB",
  "PC",        "SE",       "Saoyú-ʔehdacho National Historic Site",                 "NT",
  "PC",        "TN",       "Terra Nova National Park",                              "NL",
  "PC",        "WB",       "Wood Buffalo National Park",                            NA,
  "PC",        "WL",       "Waterton Lakes National Park",                          "AB",
  "PC",        "WP",       "Wapusk National Park",                                  "MB",
  "PC",        "YO",       "Yoho National Park",                                    "BC"
)

# Every PC and DND row must find its park or base in the lookup.
pc_dnd <- nbac_raw |>
  filter(ADMIN_NAME %in% c("PC", "DND")) |>
  distinct(ADMIN_NAME, ADMIN_DIV)
stopifnot(nrow(anti_join(pc_dnd, park_lookup, by = c("ADMIN_NAME", "ADMIN_DIV"))) == 0,
          all(park_lookup$park_prov %in% c(provinces$prov_abbr, NA)))

Wood Buffalo National Park needs a map. The script reads only its 770 polygons from inside the zip, takes one point inside each fire, and asks which province or territory contains it. 618 records fall in Alberta and 152 in the Northwest Territories.

# Read the polygons of the Wood Buffalo records only (770 of 52,610), straight
# from the zip: "/vsizip/" lets sf look inside a zip file, and the SQL query
# keeps only the rows we ask for. This takes about 20 seconds.
wb_query <- paste("SELECT GID FROM", nbac_layer, "WHERE ADMIN_DIV = 'WB'")
wb_polygons <- read_sf(file.path("/vsizip", nbac_zip, paste0(nbac_layer, ".shp")),
                       query = wb_query)

# Take one point inside each fire (st_point_on_surface) and find the
# province or territory that contains it. A few polygons are not valid
# shapes, so repair them first. The province outlines (m00) are moved into
# the NBAC map projection so that both layers use the same coordinates.
prov_shapes <- read_sf(file.path(clean_dir, "canada_provinces_2021.gpkg")) |>
  select(prov_abbr) |>
  st_transform(st_crs(wb_polygons))
wb_points <- st_sf(
  GID = wb_polygons$GID,
  geometry = st_point_on_surface(st_make_valid(st_geometry(wb_polygons)))
)
wb_prov <- st_join(wb_points, prov_shapes) |>
  st_drop_geometry() |>
  as_tibble() |>
  rename(wb_prov = prov_abbr)

stopifnot(nrow(wb_prov) == 770, !anyDuplicated(wb_prov$GID), !anyNA(wb_prov$wb_prov))
count(wb_prov, wb_prov)

Now build the clean fire table: one row per record, with its province or territory, the best start date, the official area, and the other useful fields under clearer names. The last line counts how each record got its province.

# One row per NBAC record (a fire, or the piece of a fire in one
# jurisdiction). The best start date is the agency start date when it falls
# in the fire year; otherwise it is the first satellite hotspot date. (36
# records from 2024-2025 have an agency start date in an earlier year: some
# are fires that kept burning over the winter, some look like typing errors.)
nbac_fires <- nbac_raw |>
  left_join(park_lookup, by = c("ADMIN_NAME", "ADMIN_DIV")) |>
  left_join(wb_prov, by = "GID") |>
  mutate(
    agency_start_ok = !is.na(AG_SDATE) & year(AG_SDATE) == YEAR,
    prov_abbr = case_when(
      ADMIN_NAME %in% provinces$prov_abbr ~ ADMIN_NAME,
      ADMIN_DIV %in% "WB"                 ~ wb_prov,
      .default                            = park_prov
    ),
    prov_source = case_when(
      ADMIN_NAME %in% provinces$prov_abbr ~ "agency code",
      ADMIN_DIV %in% "WB"                 ~ "polygon location",
      .default                            = "park or base"
    ),
    start_date = if_else(agency_start_ok, AG_SDATE, HS_SDATE),
    start_date_source = case_when(
      agency_start_ok   ~ "agency",
      !is.na(HS_SDATE)  ~ "hotspot",
      .default          = NA
    )
  ) |>
  arrange(YEAR, NFIREID, ADMIN_NAME, ADMIN_DIV) |>
  add_count(GID, name = "n_records") |>
  transmute(
    fire_id = GID,
    year = as.integer(YEAR),
    n_records,
    agency = ADMIN_NAME,
    park_or_base,
    prov_abbr,
    prov_source,
    start_date,
    start_date_source,
    agency_start_date = AG_SDATE,
    agency_end_date = AG_EDATE,
    hotspot_start_date = HS_SDATE,
    hotspot_end_date = HS_EDATE,
    area_ha = round(ADJ_HA, 2),
    mapped_area_ha = round(POLY_HA, 2),
    area_adjusted = ADJ_FLAG %in% "true",
    cause = FIRECAUS,
    prescribed = PRESCRIBED %in% "true",
    source = BASRC,
    map_source = FIREMAPS,
    map_method = FIREMAPM
  )

nbac_fires |> count(prov_source)

Check the grain (one row per record, no duplicated key) and the values, then write the file.

# Grain: one row per NBAC record, 52,610 rows. The key is fire_id + agency +
# park_or_base: a fire has one row per jurisdiction it burned in.
stopifnot(
  nrow(nbac_fires) == 52610,
  !anyDuplicated(select(nbac_fires, fire_id, agency, park_or_base)),
  all(nbac_fires$year %in% years),
  all(nbac_fires$prov_abbr %in% provinces$prov_abbr),
  all(nbac_fires$area_ha >= 0), all(nbac_fires$area_ha <= nbac_fires$mapped_area_ha),
  all(year(nbac_fires$start_date) == nbac_fires$year, na.rm = TRUE),
  all(nbac_fires$cause %in% c("Natural", "Human", "Undetermined"))
)
write_csv(nbac_fires, file.path(clean_dir, "nbac_fires_1972_2025.csv"), na = "")

Sum the records to province or territory × year. Wildfires and prescribed burns are summed separately, and expand_grid() makes sure that every province has every year.

# Province or territory x year. Prescribed burns are set apart. The May-
# September area uses the best start date; area with no start date is shown
# in its own column, so the gap is visible.
burns <- nbac_fires |>
  mutate(may_sep = month(start_date) %in% 5:9)

wildfire_py <- burns |>
  filter(!prescribed) |>
  summarise(
    n_fires = n_distinct(fire_id),
    burned_area_ha = sum(area_ha),
    burned_area_may_sep_ha = sum(area_ha[may_sep]),
    burned_area_no_start_ha = sum(area_ha[is.na(start_date)]),
    .by = c(prov_abbr, year)
  )
prescribed_py <- burns |>
  filter(prescribed) |>
  summarise(n_prescribed = n_distinct(fire_id), prescribed_area_ha = sum(area_ha),
            .by = c(prov_abbr, year))

# Start from every province x year, so that a year with no mapped burn is
# a row of zeros instead of a missing row.
nbac_prov_year <- expand_grid(provinces, year = years) |>
  left_join(wildfire_py, by = c("prov_abbr", "year")) |>
  left_join(prescribed_py, by = c("prov_abbr", "year")) |>
  mutate(
    across(c(n_fires, n_prescribed), ~ as.integer(replace_na(.x, 0))),
    across(ends_with("_ha"), ~ round(replace_na(.x, 0), 1))
  ) |>
  arrange(prov_abbr, year)

Check the panel: 702 rows, no duplicated province-year, no missing cell, and the same total area as the fire file. Then write it.

# Grain: one row per province or territory x year, 13 x 54 = 702 rows.
stopifnot(
  nrow(nbac_prov_year) == 13 * length(years),
  !anyDuplicated(select(nbac_prov_year, prov_abbr, year)),
  !anyNA(nbac_prov_year),
  with(nbac_prov_year, all(burned_area_may_sep_ha + burned_area_no_start_ha <=
                             burned_area_ha + 0.1)),
  abs(sum(nbac_prov_year$burned_area_ha) -
        sum(nbac_fires$area_ha[!nbac_fires$prescribed])) < 1
)
write_csv(nbac_prov_year, file.path(clean_dir, "nbac_province_year_1972_2025.csv"),
          na = "")

A first look at the result. 2023 burned 14.8 million hectares (Mha), about twice the next-largest year (2025, 7.3 Mha). In 2023, Quebec (4.26 Mha), the Northwest Territories (3.56 Mha), Alberta (2.78 Mha), and British Columbia (2.22 Mha) burned the most.

# A quick look: Canada's burned area by year, in million hectares (Mha).
# Provincial pieces of a fire do not overlap, so areas add up across
# provinces. Fire counts do not: a fire in two provinces counts in both.
nbac_national <- nbac_prov_year |>
  summarise(burned_area_mha = sum(burned_area_ha) / 1e6, .by = year)
nbac_national |>
  slice_max(burned_area_mha, n = 5)

# 2023 by province or territory, largest first.
nbac_prov_year |>
  filter(year == 2023) |>
  select(prov_abbr, n_fires, burned_area_ha, burned_area_may_sep_ha) |>
  arrange(desc(burned_area_ha))

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
nbac_fires_1972_2025.csv	52,610 (52,152 distinct fires)	an NBAC record: one fire, or the piece of a fire inside one province, territory, national park, or military base	7.0 MB
nbac_province_year_1972_2025.csv	702 (13 provinces and territories × 54 years)	a province or territory × fire year	0.03 MB

Clean variables

nbac_fires_1972_2025.csv: one row per NBAC record, sorted by year, fire number, agency, and park or base code. The key is fire_id + agency + park_or_base. Four raw columns are left out: NFIREID (it is part of fire_id), ADMIN_DIV (replaced by the park or base name), CAPDATE (the date of the source image, not a fire date), and VERSION (the same on every row).

Variable	Type	Units	Description
fire_id	chr	—	Fire identifier, the raw GID: fire year and fire number joined by _, e.g., 2023_346. Unique to one fire across all years. It repeats when a fire has several records
year	dbl	—	Fire year (raw YEAR), 1972–2025
n_records	dbl	records	Number of records of this fire in the file, 1–3. More than 1 means the fire crossed a provincial, territorial, or national park boundary
agency	chr	—	Raw ADMIN_NAME: the jurisdiction where the record is mapped. A province or territory code, PC (Parks Canada), or DND (Department of National Defence). It is not always a province: use prov_abbr for that
park_or_base	chr	—	For PC and DND records, the name of the national park, national historic site, or military base, from the raw ADMIN_DIV code; names are shortened from the metadata. Empty for province and territory records
prov_abbr	chr	—	2-letter code of the province or territory where the record lies. Never empty. Matches prov_abbr in canada_provinces_2021.csv. Records from before 1999 in today’s Nunavut are coded NT (see “Checks and cautions”)
prov_source	chr	—	How prov_abbr was found: agency code (agency is a province or territory, 50,193 records), park or base (the province of the park or base, 1,647), or polygon location (Wood Buffalo National Park: the province or territory that contains a point inside the fire’s outline, 770)
start_date	date	—	Best start date: agency_start_date if it falls in the fire year, otherwise hotspot_start_date. Empty when neither exists (10,094 records, 3,884 of them from the 1970s)
start_date_source	chr	—	Where start_date comes from: agency (40,111 records) or hotspot (2,405). Empty when start_date is empty
agency_start_date	date	—	Fire start date reported by the agency (raw AG_SDATE); the earliest date for a fire that crosses a border. Empty if not reported. 36 records from 2024–2025 have a date from an earlier year
agency_end_date	date	—	Fire end date reported by the agency (raw AG_EDATE); the latest date for a fire that crosses a border. Empty if not reported. It can fall in the next year
hotspot_start_date	date	—	Date of the first satellite hotspot inside the fire’s outline (raw HS_SDATE). Empty if there was none; always empty before 1989
hotspot_end_date	date	—	Date of the last satellite hotspot inside the fire’s outline (raw HS_EDATE). Empty if there was none
area_ha	dbl	ha	Official burned area of the record (raw ADJ_HA), 2 decimals. Use this column for burned area
mapped_area_ha	dbl	ha	Area of the polygon before any adjustment (raw POLY_HA), 2 decimals. Equal to area_ha unless area_adjusted is TRUE
area_adjusted	lgl	—	TRUE if area_ha was reduced by the area adjustment model (raw ADJ_FLAG = true, 2,579 records)
cause	chr	—	Cause of ignition (raw FIRECAUS): Natural, Human, or Undetermined
prescribed	lgl	—	TRUE for a prescribed burn reported by the agency (raw PRESCRIBED = true, 411 records)
source	chr	—	Product the outline comes from (raw BASRC): Agency, MAFiMS, C2C, or TFMS
map_source	chr	—	Data source or platform used to identify the burn (raw FIREMAPS), e.g., Landsat or Aerial survey
map_method	chr	—	Method used to draw the outline (raw FIREMAPM), e.g., Processed imagery or Buffered points

nbac_province_year_1972_2025.csv: one row per province or territory × fire year, sorted by prov_abbr and year. Every province and territory has all 54 years, and no cell is missing: a province-year with no mapped fire has zeros.

Variable	Type	Units	Description
prov_abbr	chr	—	2-letter province or territory code, as in canada_provinces_2021.csv
prov_name	chr	—	Province or territory name in English
year	dbl	—	Fire year, 1972–2025
n_fires	dbl	fires	Number of distinct wildfires (prescribed burns excluded) with at least one record in this province and year. A fire that burned in two provinces counts in both, so do not add these counts across provinces
burned_area_ha	dbl	ha	Total area_ha of the wildfire records (prescribed burns excluded), 1 decimal. Adds up across provinces
burned_area_may_sep_ha	dbl	ha	The part of burned_area_ha from records whose start_date falls in May–September. It is the area of fires that started in those months, not the area that burned during them
burned_area_no_start_ha	dbl	ha	The part of burned_area_ha from records with no start_date. These fires may have started in any month
n_prescribed	dbl	fires	Number of distinct prescribed burns with a record in this province and year
prescribed_area_ha	dbl	ha	Total area_ha of the prescribed burns, 1 decimal

Checks and cautions

• Validation script. R/check_m10.R (not part of the student path) runs every check below. It reads the raw zip, the clean files, and the research project’s copies of the raw files.
• Raw copies. The three raw files are byte-identical (same MD5 checksum) to the research project’s copies, which were downloaded from the CWFIS download folder on 2026-09-27. The PDF inside the zip is the same file as the metadata PDF.
• Two readers agree. foreign::read.dbf() and GDAL (sf::read_sf()) read the same 52,610 × 19 table, and all 19 columns agree. One difference: GDAL turns an empty dBASE date into the impossible date -001-11-30 instead of NA (36,487 empty HS_SDATE values, for example). read.dbf() returns NA, which is one reason the script uses it.
• Independent rebuild. A second build in base R, using provinces taken from the polygons themselves, reproduces every column of the fire file exactly (areas within the 0.005 ha rounding). In the province-year file, n_fires and n_prescribed match exactly, and the areas differ by at most 1.2 ha, because the script sums areas that were already rounded to 0.01 ha.
• Provinces checked on the map. For every Parks Canada and Defence record (2,417) and every record of 1995, 2014, and 2023 (4,056 province and territory records), a point inside the polygon was placed on the provincial map. It falls in the assigned province or territory for 2,416 of the 2,417 park and base records (the point of 2005_1352 falls outside every provincial outline) and for 4,042 of the 4,056 other records. Of the remaining 14, 13 points fall outside every provincial outline, and one 1995 record coded NT lies in today’s Nunavut (see the next point). All the records of each park or base lie in one province.
• Nunavut before 1999. Nunavut was split from the Northwest Territories in 1999. NBAC codes a record by the territory of its time: there is no NU record before 1999. 94 of the 2,350 NT records from 1972–1998 lie in today’s Nunavut: 281,617 ha of 14.6 Mha (1.9%). So NU is 0 in 1972–1998 by construction. For a consistent series over time, add NT and NU together.
• Wood Buffalo National Park. Its 770 records go to Alberta (618) or the Northwest Territories (152). For all 770, the province that contains the point is also the province that holds the largest part of the fire. Only 11 records have less than 99% of their area in one province (the lowest share is 54%). Splitting those fires by area instead would move about 18,000 ha from the Northwest Territories to Alberta over 1972–2025, about 0.5% of the park’s 3.4 Mha.
• Published totals: National Forestry Database. The National Forestry Database (NFD) holds the area burned that each agency reports. Its regional totals for 1990–2024 are published in Table A.3 of Environment and Climate Change Canada (2025), Canadian Environmental Sustainability Indicators: Forest management and disturbances (source: Canadian Council of Forest Ministers, National Forestry Database, 2025), in km² and without Nunavut. Without Nunavut, NBAC’s national area is 77% to 111% of the NFD figure in each year (median 91%); over 1990–2024 it sums to 87% of the NFD total, and the two series correlate at 0.998 across years. By region, NBAC is 103% of NFD in Quebec, 121% in the Atlantic provinces, 86% in the Prairies, 85% in the territories, 82% in British Columbia, and 80% in Ontario. One reason NBAC is usually lower: outlines that agencies draw by hand include unburned patches inside the fire, while satellite outlines and the area adjustment leave them out (Skakun et al. 2021, Environmental Research Letters 16: 064014).
• 2023. NBAC gives 14.80 Mha for 2023 (14.79 Mha without Nunavut), the largest year since 1972. The NFD gives 17.61 Mha (176,065 km², without Nunavut). Jain et al. (2024, Nature Communications 15, doi:10.1038/s41467-024-51154-7) report about 15 Mha for the record 2023 season. The 7 prescribed burns of 2023 match the 7 excluded in the Lecture 6 map extract.
• No other project file to compare. The research project has no cleaned file built from NBAC, so this module’s outputs are checked against the raw data and the published totals above, not against a project build.
• Fire counts do not add up across provinces. A fire that burned in two provinces counts once in each. Summed over all province-years, n_fires is 335 higher than the number of distinct wildfires. For national counts, count distinct fire_id in the fire file. Areas do add up: the pieces of a fire do not overlap.
• Early years have fewer dates and fewer small fires. Before 1989 there are no satellite hotspots, so start_date comes only from agencies, and many records have none: more than 10% of the area burned has no start date in 1972–1980, 1984, 1985, and 1988. Use burned_area_may_sep_ha with care before 1989. From 1989 on, 78% to 100% of each year’s area comes from fires that started in May–September (more than 90% in 35 of 37 years; the exceptions are 2006 and 2024). Fire counts are also hard to compare across decades: the metadata’s version history says that agency fires under 10 ha were added only from 2004 on, so later years include more small fires.
• Dates that cross years. 36 records from 2024–2025 have an agency start date in an earlier year. Some are fires that smoldered through the winter (holdover fires); others look like typing errors. For these, start_date falls back to the hotspot date, or stays empty. agency_end_date falls in the next year for 75 records.
• Possible double mapping. Two pairs of records share the same year, province, area, and dates (1985_326 and 1985_327; 2014_2163 and 2014_2242). Their outlines do not overlap (the two fires of each pair are 25.5 km and 1,108 km apart), so all four are kept as separate fires.
• Tiny records. 213 records are smaller than 0.005 ha and show area_ha = 0.00 after rounding; most are buffered points or GPS traces. They still count in n_fires.
• Area adjustment is small in this release. 2,579 agency records have an adjusted area; on those records area_ha is 71% to 100% of mapped_area_ha. Over all records, the adjustment removes 0.59% of the mapped area.
• Burned area is not smoke. NBAC tells you where and how much burned in a year, not when the fire grew or how much smoke it made. For timing, use the daily fire detections (FIRMS) and the smoke data. The annual totals are revised when new releases come out; cite the release (20260513).
• Need the outlines? The clean files have no geometry. To map fires, read the polygons from the zip with sf::read_sf() and an SQL query, as in chunk m10-wood-buffalo, or download one year at a time (NBAC_<year>_20260513.zip) from the CWFIS download folder. All years at once need several GB of memory.
• Reading the files. read_csv() guesses every column type correctly (fire_id as text, dates as dates, area_adjusted and prescribed as logical), but you can still give col_types. Empty cells mean “not reported” for dates and “not a park or base” for park_or_base.
• Memory and time. The student script runs in about 1 minute, with a peak memory use of about 0.6 GB. The validation script takes about 4 to 6 minutes and peaks at about 1.7 GB.

NOAA HMS smoke plumes

The Hazard Mapping System (HMS) is a daily map of fire and smoke over North America made by NOAA’s Office of Satellite and Product Operations (OSPO). Trained analysts look at visible daytime images from the GOES weather satellites and draw an outline (a polygon) around every smoke plume they can see. Each outline records the satellite, the start and end time of the images used, and, from 2007–2010 on, a density class: Light, Medium, or Heavy. NOAA publishes one zipped shapefile per year with every outline drawn that year.

In this project HMS answers the question “was there smoke over this county on this day?”, and it reaches back to August 2005 with one consistent method on both sides of the border. It is the screen that other smoke products (Childs et al., CSU) use to decide which days are smoke days. It cannot tell you how much smoke people breathed. Analysts see smoke anywhere in the column of air above the ground, and in the eastern United States Canadian smoke often travels high aloft. A plume marks smoke overhead, not smoke at breathing height. The density class is a rough visual judgment of thickness, not a measured concentration.

Provider	NOAA/NESDIS Office of Satellite and Product Operations (OSPO), Hazard Mapping System
Coverage	plumes 5 August 2005 – 31 December 2025 (UTC dates) · North America and nearby oceans (across all years, outlines reach from 180° W to 24° E and from 1° S to 87° N; the largest 2023–2025 plumes reach Europe)
Raw unit	one polygon = one smoke plume drawn by an analyst from one set of satellite images
Clean unit	plume table: one row = one plume polygon; calendar: one row = one UTC calendar day; county-day: one row = one county × one UTC day with at least one plume (sparse); county-month: one row = one county × one month (complete)
Licence and citation	U.S. Government work, public domain. Cite as NOAA/NESDIS OSPO, Hazard Mapping System Fire and Smoke Product, smoke polygons (annual bundles, accessed 27 September 2026). Method: Ruminski, M., Kondragunta, S., Draxler, R., and Zeng, J. (2006), “Recent changes to the Hazard Mapping System,” 15th International Emission Inventory Conference, U.S. EPA
Documentation	HMS product page (OSPO) · annual shapefile bundles

Raw files

File (in wildfire_data/raw/)	Size	Format	One row is
hms/hms_smoke2005__v20250204.zip	0.66 MB	zip archive: one ESRI shapefile hms_smoke2005.shp + .shx, .dbf, .prj	one smoke polygon (6,289 polygons)
hms/hms_smoke2006__v20250204.zip	1.81 MB	zip archive: one ESRI shapefile hms_smoke2006.shp + .shx, .dbf, .prj	one smoke polygon (15,443 polygons)
hms/hms_smoke2007__v20250204.zip	2.32 MB	zip archive: one ESRI shapefile hms_smoke2007.shp + .shx, .dbf, .prj	one smoke polygon (19,872 polygons)
hms/hms_smoke2008__v20250204.zip	2.71 MB	zip archive: one ESRI shapefile hms_smoke2008.shp + .shx, .dbf, .prj	one smoke polygon (23,277 polygons)
hms/hms_smoke2009__v20250204.zip	2.51 MB	zip archive: one ESRI shapefile hms_smoke2009.shp + .shx, .dbf, .prj	one smoke polygon (23,473 polygons)
hms/hms_smoke2010__v20250204.zip	2.91 MB	zip archive: one ESRI shapefile hms_smoke2010.shp + .shx, .dbf, .prj	one smoke polygon (27,230 polygons)
hms/hms_smoke2011__v20250204.zip	3.88 MB	zip archive: one ESRI shapefile hms_smoke2011.shp + .shx, .dbf, .prj	one smoke polygon (33,720 polygons)
hms/hms_smoke2012__v20250204.zip	3.28 MB	zip archive: one ESRI shapefile hms_smoke2012.shp + .shx, .dbf, .prj	one smoke polygon (27,966 polygons)
hms/hms_smoke2013__v20250204.zip	2.63 MB	zip archive: one ESRI shapefile hms_smoke2013.shp + .shx, .dbf, .prj	one smoke polygon (23,151 polygons)
hms/hms_smoke2014__v20250204.zip	2.00 MB	zip archive: one ESRI shapefile hms_smoke2014.shp + .shx, .dbf, .prj	one smoke polygon (18,562 polygons)
hms/hms_smoke2015__v20250204.zip	1.69 MB	zip archive: one ESRI shapefile hms_smoke2015.shp + .shx, .dbf, .prj	one smoke polygon (16,349 polygons)
hms/hms_smoke2016__v20250204.zip	2.25 MB	zip archive: one ESRI shapefile hms_smoke2016.shp + .shx, .dbf, .prj	one smoke polygon (21,267 polygons)
hms/hms_smoke2017__v20250204.zip	5.19 MB	zip archive: one ESRI shapefile hms_smoke2017.shp + .shx, .dbf, .prj	one smoke polygon (26,187 polygons)
hms/hms_smoke2018__v20250204.zip	7.83 MB	zip archive: one ESRI shapefile hms_smoke2018.shp + .shx, .dbf, .prj	one smoke polygon (41,294 polygons)
hms/hms_smoke2019__v20250204.zip	7.70 MB	zip archive: one ESRI shapefile hms_smoke2019.shp + .shx, .dbf, .prj	one smoke polygon (42,951 polygons)
hms/hms_smoke2020__v20250204.zip	8.51 MB	zip archive: one ESRI shapefile hms_smoke2020.shp + .shx, .dbf, .prj	one smoke polygon (45,287 polygons)
hms/hms_smoke2021__v20250204.zip	5.88 MB	zip archive: one ESRI shapefile hms_smoke2021.shp + .shx, .dbf, .prj	one smoke polygon (27,445 polygons)
hms/hms_smoke2022__v20250204.zip	5.45 MB	zip archive: one ESRI shapefile hms_smoke2022.shp + .shx, .dbf, .prj	one smoke polygon (21,768 polygons)
hms/hms_smoke2023__v20250204.zip	6.27 MB	zip archive: one ESRI shapefile hms_smoke2023.shp + .shx, .dbf, .prj	one smoke polygon (20,191 polygons)
hms/hms_smoke2024__v20250409.zip	5.05 MB	zip archive: one ESRI shapefile hms_smoke2024.shp + .shx, .dbf, .prj	one smoke polygon (12,527 polygons)
hms/hms_smoke2025__v20260101.zip	8.98 MB	zip archive: one ESRI shapefile hms_smoke2025.shp + .shx, .dbf, .prj	one smoke polygon (24,238 polygons)

In all, the 21 files take 89.5 MB and hold 518,487 polygons. Inside each zip, .shp holds the outlines, .shx is an index, .dbf is the attribute table, and .prj gives the coordinate system (WGS 84 longitude/latitude).

NOAA posts each year as hms_smoke<year>.zip. The research project added __v<date> when it downloaded the files on 27 September 2026: the date NOAA last posted the file. It is v20250204 for 2005–2023, v20250409 for 2024, and v20260101 for 2025. NOAA regenerated its whole archive when it upgraded HMS in 2022, so older years are re-posted from time to time; check this date if you download a fresh copy. The project also has a partial 2026 file, which is not shipped.

Raw variables

All 21 shapefiles have the same four attributes, all stored as text in the .dbf file, plus the outline itself. Descriptions follow the OSPO product page; counts are over all 518,487 polygons.

Variable	Type	Description
Satellite	text (15 characters)	Satellite whose images the analyst used. GOES (2005 – 31 October 2016; the provider does not say which GOES), then GOES-EAST or GOES-WEST (from 20 October 2016). Five plumes in 2024–2025 name a polar-orbiting satellite: NOAA 20, NOAA 21, or Suomi NPP
Start	text (12 characters), YYYYDDD HHMM	Time of the first image used for the plume, in UTC (Coordinated Universal Time, the clock at Greenwich, 4 to 8 hours ahead of U.S. clocks). YYYY is the year, DDD the day of the year (001 = 1 January), HHMM hours and minutes. Example: 2023156 1300 = 5 June 2023, 13:00 UTC
End	text (12 characters), YYYYDDD HHMM	Time of the last image used for the plume, in UTC, same format. Equal to Start for 41,084 of the 518,487 raw polygons (drawn from one image)
Density	text (6 characters)	Analyst’s judgment of how thick the smoke looks: Light, Medium, or Heavy; the text NA when the plume has no label. Labels are missing for all plumes in 2005–2006 and for most in 2007, 2009, and early 2010 (see “Checks and cautions”). The OSPO page says the classes were stored as numbers before July 2022 (it lists them as 5, 16, 21); Liu et al. (2024; full reference under “Checks and cautions”) report them as 5, 16, and 27 µg/m³, rough smoke PM2.5 equivalents that the 2022 update dropped. The archive NOAA posts today uses the words in every year
geometry	polygon	Outline of the plume, longitude/latitude in degrees (WGS 84). One ring per plume, without holes. 751 outlines (in 2009, 2010, 2012, and 2015) are not closed, and 90 are points, lines, or flat shapes with no area

Cleaning steps

The code makes these decisions:

1. Day = UTC date of the plume’s start time. This is the research project’s convention and the way NOAA dates its daily analysis. About 4% of plumes start between 00:00 and 04:59 UTC, which is still the evening of the previous day in U.S. time; they count for the next day.
2. Repair the outlines, then drop the ones with no area. Open rings are closed, st_make_valid() fixes self-crossing outlines, and 90 of 518,487 polygons that are single points, lines, or flat shapes are dropped.
3. Flat geometry in longitude/latitude. sf_use_s2(FALSE) makes sf treat longitude and latitude as flat x and y, as the project does. It is faster and handles the hand-drawn outlines; at county scale the answer is practically the same as on a sphere.
4. Counties are land areas. The county outlines from module m00 are clipped to the shoreline, so a plume over Lake Erie or the Atlantic does not count for a coastal county.
5. Population shares use block-group centers. Each Census block group (a neighborhood of roughly 600 to 3,000 people) is represented by its 2020 center of population. A county’s share under a plume is the population of the centers inside the plume divided by the county’s population.
6. County panels start in 2006. HMS starts on 5 August 2005, so 2005 is in the plume table and calendar but not in the county panels.
7. Density code 0 = plume without a label. In the county files, max_density is the densest plume over the county that day: 0 unlabeled, 1 Light, 2 Medium, 3 Heavy.
8. Write the county-day file one year at a time. The full county-day table has about 4 million rows. Each year’s rows are checked, appended to the end of the file, and summarized by month right away, so the whole table is never in memory at once.

First, turn off spherical geometry and set the years.

# HMS plume outlines are stored in longitude/latitude. Like the research
# project, we test overlaps with flat (planar) geometry, not with sf's default
# spherical engine (s2). Planar tests are much faster here, and they accept the
# hand-drawn outlines once we repair them. At county scale the two engines
# give practically the same answer.
sf_use_s2(FALSE)

hms_dir <- file.path(raw_dir, "hms")
years <- 2005:2025        # plume table and calendar
panel_years <- 2006:2025  # county panels: the 2005 file starts on 5 August

dir.create(clean_dir, recursive = TRUE, showWarnings = FALSE)

List the 21 yearly zip files and check there is exactly one per year.

# One zipped shapefile per year. The end of the name is the date NOAA posted
# the file (v20250204 = 4 February 2025). NOAA re-posts a year when it
# revises it, so check this date when you download a newer copy.
hms_files <- tibble(
  path = list.files(hms_dir, pattern = "^hms_smoke\\d{4}__v\\d{8}\\.zip$",
                    full.names = TRUE)
) |>
  mutate(year = as.integer(str_sub(basename(path), 10, 13)))

# Exactly one file for each year.
stopifnot(setequal(hms_files$year, years), !anyDuplicated(hms_files$year))
hms_files

The reading function opens the shapefile straight from the zip (GDAL’s /vsizip/ path), repairs the outlines, and converts the times and density labels.

# 751 outlines (in 2009, 2010, 2012, and 2015) are "open": the last corner
# is not a copy of the first. GDAL warns "Non closed ring detected",
# and GEOS, the geometry engine behind sf, refuses to work with an open ring.
# close_ring() repeats the first corner at the end of the ring when needed.
close_ring <- function(xy) {
  if (any(xy[1, ] != xy[nrow(xy), ])) xy <- rbind(xy, xy[1, ])
  xy
}

read_hms_year <- function(year) {
  shp <- file.path("/vsizip", hms_files$path[hms_files$year == year],
                   str_glue("hms_smoke{year}.shp"))
  # Silence only the open-ring warning, because close_ring() repairs those
  # rings below. Any other warning still shows.
  raw <- withCallingHandlers(
    read_sf(shp),
    warning = function(w) {
      if (str_detect(conditionMessage(w), "Non closed ring")) {
        invokeRestart("muffleWarning")
      }
    }
  )
  # Every outline is one ring. The .prj file says WGS 84 longitude/latitude,
  # which is EPSG:4326.
  closed <- map(st_geometry(raw), \(poly) st_polygon(map(poly, close_ring)))
  st_geometry(raw) <- st_sfc(closed, crs = 4326)

  raw |>
    mutate(
      # plume_id = year + row number in the raw file, so you can find the
      # plume again in the shapefile.
      plume_id = str_c(year, "_", str_pad(row_number(), 5, pad = "0")),
      # A ring needs 4 points (3 corners + the first one again) to enclose
      # an area. 77 outlines are single points or there-and-back lines.
      n_points = map_int(geometry, \(poly) nrow(poly[[1]]))
    ) |>
    filter(n_points >= 4) |>
    st_make_valid() |>
    # Repair turns 13 flat outlines (all corners on one line) into lines.
    # A line has no area, so keep polygons only.
    filter(st_geometry_type(geometry) %in% c("POLYGON", "MULTIPOLYGON")) |>
    mutate(
      # Start and End are "YYYYDDD HHMM": year, day of the year, and the
      # hour and minute in UTC (Coordinated Universal Time, the clock at
      # Greenwich, 4 to 8 hours ahead of U.S. clocks).
      start_utc = as.POSIXct(Start, format = "%Y%j %H%M", tz = "UTC"),
      end_utc = as.POSIXct(End, format = "%Y%j %H%M", tz = "UTC"),
      # The project's day: the UTC date of the plume's start time.
      date = as_date(start_utc),
      satellite = str_trim(Satellite),
      # Unlabeled plumes store the text "NA"; make it a real missing value.
      density = na_if(str_trim(Density), "NA"),
      # Position in the list = code: Light 1, Medium 2, Heavy 3; unlabeled 0.
      density_code = coalesce(match(density, c("Light", "Medium", "Heavy")), 0L)
    ) |>
    select(plume_id, date, start_utc, end_utc, satellite, density, density_code)
}

Try the function on one year and look at the result before running all years.

# Look at one year before running all of them. 2023 is the year of the
# eastern Canadian fires.
plumes_2023 <- read_hms_year(2023)
plumes_2023

# Every time parses, and every plume starts in its file's year.
stopifnot(!anyNA(plumes_2023$start_utc), !anyNA(plumes_2023$end_utc),
          all(year(plumes_2023$date) == 2023),
          all(plumes_2023$end_utc >= plumes_2023$start_utc))

count(st_drop_geometry(plumes_2023), satellite, density)

The plume table keeps the attributes, the area in km², and the center of each plume, computed in an equal-area projection.

# One row per plume, without its outline. Area and center are computed in
# CONUS Albers (EPSG:5070), an equal-area map projection in meters, so
# st_area() gives true areas. The center is moved back to longitude/latitude.
plume_table <- function(plumes) {
  outlines <- st_transform(st_geometry(plumes), 5070)
  centers <- outlines |>
    st_centroid() |>
    st_transform(4326) |>
    st_coordinates()
  plumes |>
    st_drop_geometry() |>
    transmute(
      plume_id, date, start_utc, end_utc, satellite, density,
      area_km2 = round(as.numeric(st_area(outlines)) / 1e6, 1),  # m² to km²
      centroid_lon = round(centers[, "X"], 4),
      centroid_lat = round(centers[, "Y"], 4)
    )
}

plume_table(plumes_2023)

Read the county outlines and block-group population centers written by module m00.

# County outlines from module m00 are clipped to the shoreline, so a plume
# that covers only the Great Lakes or the ocean does not count for a county.
counties <- read_sf(file.path(clean_dir, "counties_2020.gpkg")) |>
  select(GEOID)

# Block-group centers of population stand for where people live. Block
# groups with no residents add nothing to a population share, so drop them.
bg_points <- read_csv(
  file.path(clean_dir, "census_bg_centers_2020.csv"),
  col_types = cols(bg_geoid = col_character(), GEOID = col_character(),
                   pop_2020 = col_double(), lat = col_double(), lon = col_double())
) |>
  filter(pop_2020 > 0) |>
  st_as_sf(coords = c("lon", "lat"), crs = 4326)

county_pop <- bg_points |>
  st_drop_geometry() |>
  summarize(county_pop = sum(pop_2020), .by = GEOID)

stopifnot(nrow(counties) == 3108, nrow(county_pop) == 3108,
          setequal(county_pop$GEOID, counties$GEOID))

Find which plumes touch which county, using st_intersects() and a lookup instead of a full spatial join.

# Which plumes touch which county? st_intersects(plumes, counties) answers
# with a list: for plume 1, the row numbers of the counties it touches; then
# plume 2; and so on. as.data.frame() turns that list into one row per
# (plume, county) pair: row.id is the plume's row and col.id the county's
# row. We look up each pair's GEOID, date, and density, then count plumes
# per county and day and keep the densest one. (st_join() gives the same
# pairs but copies a county outline for every pair, which is slow and needs
# much more memory.) sf prints a note that it treats longitude/latitude as
# flat; we chose that above, so the note is silenced.
county_overlap <- function(plumes) {
  pairs <- suppressMessages(st_intersects(plumes, counties)) |>
    as.data.frame()
  tibble(
    GEOID = counties$GEOID[pairs$col.id],
    date = plumes$date[pairs$row.id],
    density_code = plumes$density_code[pairs$row.id]
  ) |>
    summarize(n_plumes = n(), max_density = max(density_code),
              .by = c(GEOID, date))
}

county_overlap(plumes_2023) |>
  arrange(desc(n_plumes))

Find which population centers lie under a plume.

# Which block-group centers lie inside a plume? Same idea, with points:
# row.id is the plume's row and col.id the center's row. A center under
# several plumes on the same day keeps the densest one: sort the densest
# first, and distinct() keeps the first row of each center-day.
# (summarize(max()) gives the same answer but is slow and needs a lot of
# memory with millions of tiny groups.) Then we add up the population of
# the centers per county-day. `points` is the set of centers to test.
pop_under_plumes <- function(plumes, points) {
  pairs <- suppressMessages(st_intersects(plumes, points)) |>
    as.data.frame()
  tibble(
    bg = pairs$col.id,
    date = plumes$date[pairs$row.id],
    density_code = plumes$density_code[pairs$row.id]
  ) |>
    arrange(desc(density_code)) |>
    distinct(bg, date, .keep_all = TRUE) |>
    mutate(GEOID = points$GEOID[bg], pop = points$pop_2020[bg]) |>
    summarize(
      pop_any = sum(pop),
      pop_medium_plus = sum(pop[density_code >= 2]),
      pop_heavy = sum(pop[density_code == 3]),
      .by = c(GEOID, date)
    )
}

Combine the two into one row per county and day. To keep memory low, county_day() works through each year in 5-day windows. Inside a window it tests the population centers one day at a time, and only those in counties a plume touches. It returns the rows sorted by date and county.

# One row per county and day on which at least one plume touches the
# county's land. Population shares come from the block-group centers. A
# plume can touch the edge of a county and miss every center, so a share
# can be 0 on a plume day.
county_day_window <- function(plumes) {
  overlap <- county_overlap(plumes)
  # Test the centers one day at a time, and only those in counties that a
  # plume touches that day: other centers cannot count. A smoky day can
  # still give a million (plume, center) pairs; a whole window at once would
  # give several million.
  shares <- map(unique(plumes$date), \(d) {
    points <- bg_points[bg_points$GEOID %in% overlap$GEOID[overlap$date == d], ]
    pop_under_plumes(plumes[plumes$date == d, ], points)
  }) |>
    list_rbind()

  # left_join() keeps only county-days in `overlap`. A center in water just
  # off the coast can sit under a plume that misses the county's land; that
  # county-day is not in `overlap`, so the plume does not count for it.
  overlap |>
    left_join(shares, by = c("GEOID", "date")) |>
    left_join(county_pop, by = "GEOID") |>
    mutate(
      across(c(pop_any, pop_medium_plus, pop_heavy), \(x) replace_na(x, 0)),
      pop_share_any = round(pop_any / county_pop, 4),
      pop_share_medium_plus = round(pop_medium_plus / county_pop, 4),
      pop_share_heavy = round(pop_heavy / county_pop, 4)
    ) |>
    select(GEOID, date, n_plumes, max_density,
           pop_share_any, pop_share_medium_plus, pop_share_heavy)
}

# In a smoky year a center can sit under plumes on a hundred days: 2020
# gives about 26 million (plume, center) pairs. So we work through the year
# in 5-day windows (days 1-5 of the year, 6-10, ...) to keep memory low.
# (Testing the 3,108 county outlines is slow, so we do that once per window,
# not once per day.) Every plume belongs to one date, so no county-day is
# split across windows.
county_day <- function(plumes) {
  window <- (yday(plumes$date) - 1) %/% 5
  map(sort(unique(window)), \(w) county_day_window(plumes[window == w, ])) |>
    list_rbind() |>
    arrange(date, GEOID)
}

The county-day panel is checked one year at a time, inside the main loop. This function holds the grain checks.

# SPARSE panel. Grain: one row per county and day with at least one plume
# over the county's land. A missing county-day means no plume touched the
# county that day, which counts as "no smoke seen" only if hms_calendar
# says the day was analysed. We check one year at a time, inside the main
# loop. A key (GEOID, date) cannot repeat across years, so unique keys in
# every year mean unique keys in the whole file.
check_county_day <- function(cd, year) {
  stopifnot(
    nrow(distinct(cd, GEOID, date)) == nrow(cd),
    all(cd$GEOID %in% counties$GEOID),
    all(between(cd$date, make_date(year, 1, 1), make_date(year, 12, 31))),
    all(cd$n_plumes >= 1),
    all(cd$max_density %in% 0:3),
    all(between(cd$pop_share_any, 0, 1)),
    all(cd$pop_share_medium_plus <= cd$pop_share_any),
    all(cd$pop_share_heavy <= cd$pop_share_medium_plus)
  )
}

The county-month panel needs only monthly counts, so each year’s county-days are summarized right away.

# For the county-month panel we only need, per county and month, how many
# days had a plume. Counting right after each year is built means we never
# have to hold all 4 million county-days in memory at once.
month_counts <- function(cd) {
  cd |>
    mutate(year = year(date), month = month(date)) |>
    summarize(
      plume_days = n(),
      medium_plus_days = sum(max_density >= 2),
      heavy_days = sum(max_density == 3),
      pop_weighted_plume_days = round(sum(pop_share_any), 3),
      .by = c(GEOID, year, month)
    )
}

Run all years. This loop takes about 15 minutes. It writes the sparse county-day file year by year, appending each year to the end of the file, so the 4 million county-days are never in memory at once.

# The main loop takes about 15 minutes. For each year it reads the plumes and
# keeps the plume table. From 2006 on it also builds the county-day rows,
# checks them, adds them to the end of the county-day file, and keeps only
# their monthly counts. Writing the county-day rows year by year keeps
# memory low: the full table has about 4 million rows. Results go into
# lists made in advance, one slot per year. The 2023 demo object is no
# longer needed, so free its memory first.
rm(plumes_2023)
county_day_file <- file.path(clean_dir, "hms_county_day_2006_2025.csv.gz")
plume_parts <- vector("list", length(years))
month_parts <- vector("list", length(years))
county_day_rows <- integer(length(years))

for (i in seq_along(years)) {
  plumes <- read_hms_year(years[i])
  stopifnot(!anyNA(plumes$start_utc), !anyNA(plumes$end_utc),
            all(year(plumes$date) == years[i]))
  plume_parts[[i]] <- plume_table(plumes)
  if (years[i] %in% panel_years) {
    cd <- county_day(plumes)
    check_county_day(cd, years[i])
    # The first panel year starts a new file (with column names); later
    # years are appended below it. read_csv() reads the result as one file.
    write_csv(cd, county_day_file, na = "", append = years[i] > panel_years[1])
    month_parts[[i]] <- month_counts(cd)
    county_day_rows[i] <- nrow(cd)
    rm(cd)
  }
  message(years[i], ": ", nrow(plumes), " plumes, ",
          county_day_rows[i], " county-days")
  # Hand the memory of this year's plumes back before reading the next year.
  rm(plumes)
  gc()
}

hms_plumes <- list_rbind(plume_parts)
plume_months <- list_rbind(month_parts)
rm(plume_parts, month_parts)
# Total county-day rows written. The file is about 110 MB as plain text, so
# it is compressed (.csv.gz); read_csv() reads .csv.gz files directly.
sum(county_day_rows)

Check and write the plume table.

# Grain: one row per plume polygon; plume_id is unique. Every plume from
# 2 June 2010 on has a density label. Before that, most plumes have none
# (see "Checks and cautions").
stopifnot(
  !anyDuplicated(hms_plumes$plume_id),
  all(between(hms_plumes$date, as_date("2005-08-05"), as_date("2025-12-31"))),
  all(hms_plumes$density %in% c("Light", "Medium", "Heavy", NA)),
  !anyNA(hms_plumes$satellite),
  all(hms_plumes$area_km2 >= 0),
  all(between(hms_plumes$centroid_lat, -90, 90)),
  all(between(hms_plumes$centroid_lon, -180, 180))
)
count(hms_plumes, year = year(date))

write_csv(hms_plumes, file.path(clean_dir, "hms_smoke_plumes_2005_2025.csv"), na = "")

Build the calendar of days with and without plumes.

# One row per calendar day, 2005-2025. A day with no plume is ambiguous:
# either the analysts saw no smoke, or nobody analysed that day (the record
# starts on 5 August 2005, and a few days later on are missing). We cannot
# tell the two apart, so "analysed" only means "at least one plume".
hms_calendar <- tibble(date = seq(as_date("2005-01-01"), as_date("2025-12-31"),
                                  by = "day")) |>
  left_join(
    hms_plumes |>
      summarize(n_plumes = n(), share_unlabeled = round(mean(is.na(density)), 3),
                .by = date),
    by = "date"
  ) |>
  mutate(n_plumes = replace_na(n_plumes, 0L), analysed = n_plumes > 0) |>
  select(date, n_plumes, analysed, share_unlabeled)

# Grain: one row per day; share_unlabeled is missing exactly on no-plume days.
stopifnot(
  nrow(hms_calendar) == 7670, !anyDuplicated(hms_calendar$date),
  sum(hms_calendar$n_plumes) == nrow(hms_plumes),
  all(is.na(hms_calendar$share_unlabeled) == !hms_calendar$analysed)
)
count(hms_calendar, year = year(date), analysed) |>
  pivot_wider(names_from = analysed, values_from = n, names_prefix = "analysed_")

write_csv(hms_calendar, file.path(clean_dir, "hms_calendar_2005_2025.csv"), na = "")

Build the complete county-month panel from the monthly counts and the calendar.

# COMPLETE panel: every county x every month, 2006-2025. The monthly counts
# come from the sparse county-day rows (plume_months, built in the loop), so
# a county-month with no plume day gets 0.
month_days <- hms_calendar |>
  filter(year(date) %in% panel_years) |>
  mutate(year = year(date), month = month(date)) |>
  summarize(analysed_days = sum(analysed), n_days = n(), .by = c(year, month))

hms_county_month <- expand_grid(GEOID = counties$GEOID, month_days) |>
  left_join(plume_months, by = c("GEOID", "year", "month")) |>
  mutate(
    across(c(plume_days, medium_plus_days, heavy_days), \(x) replace_na(x, 0L)),
    pop_weighted_plume_days = replace_na(pop_weighted_plume_days, 0)
  ) |>
  select(GEOID, year, month, plume_days, medium_plus_days, heavy_days,
         pop_weighted_plume_days, analysed_days, n_days) |>
  arrange(GEOID, year, month)

# Grain: 3,108 counties x 240 months, no missing cells. Every month in
# 2006-2025 has analysed days, so a 0 is a real zero, not a gap.
stopifnot(
  nrow(hms_county_month) == 3108 * 240,
  nrow(distinct(hms_county_month, GEOID, year, month)) == nrow(hms_county_month),
  !anyNA(hms_county_month),
  all(hms_county_month$analysed_days > 0),
  all(hms_county_month$plume_days <= hms_county_month$analysed_days),
  all(hms_county_month$heavy_days <= hms_county_month$medium_plus_days),
  all(hms_county_month$medium_plus_days <= hms_county_month$plume_days),
  all(hms_county_month$pop_weighted_plume_days <= hms_county_month$plume_days)
)

write_csv(hms_county_month, file.path(clean_dir, "hms_county_month_2006_2025.csv"),
          na = "")

When you come back to these files later, read them with explicit column types.

# How to read the clean files later. Give col_types for the plume table:
# read_csv() guesses types from the first 1,000 rows, and those (August 2005)
# have no density label, so it would guess "logical" and turn every later
# "Light"/"Medium"/"Heavy" into NA. GEOID must stay text to keep its
# leading zero (01001, not 1001). First drop the copies built above, so the
# file and the in-memory copy are not both held at once.
rm(hms_plumes, hms_county_month, plume_months)
invisible(gc())
hms_plumes <- read_csv(
  file.path(clean_dir, "hms_smoke_plumes_2005_2025.csv"),
  col_types = cols(plume_id = col_character(), density = col_character(),
                   satellite = col_character(), date = col_date())
)
hms_county_month <- read_csv(
  file.path(clean_dir, "hms_county_month_2006_2025.csv"),
  col_types = cols(GEOID = col_character())
)
stopifnot(nrow(hms_plumes) == sum(hms_calendar$n_plumes),
          sum(!is.na(hms_plumes$density)) > 400000,
          nrow(hms_county_month) == 3108 * 240)
count(hms_plumes, density)

Check the county-day file without loading all of it: read it in pieces of 500,000 rows and count the rows per year.

# In your own work, read the county-day file in one go, in a fresh R session
# (it needs about 1 GB of memory):
#   read_csv(county_day_file,
#            col_types = cols(GEOID = col_character(), date = col_date()))
# Here, after the big loop, we only check it. read_csv_chunked() reads
# 500,000 rows at a time and keeps just a count of rows per year from each
# piece. Each year must hold exactly the rows the loop appended for it.
rows_per_year <- read_csv_chunked(
  county_day_file,
  DataFrameCallback$new(\(piece, pos) count(piece, year = year(date))),
  chunk_size = 500000,
  col_types = cols(GEOID = col_character(), date = col_date())
) |>
  summarize(n = sum(n), .by = year)

stopifnot(nrow(rows_per_year) == length(panel_years),
          all(rows_per_year$year == panel_years),
          all(rows_per_year$n == county_day_rows[years %in% panel_years]))
rows_per_year

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
hms_smoke_plumes_2005_2025.csv	518,397	one plume polygon, without its outline	52.1 MB
hms_calendar_2005_2025.csv	7,670	one UTC calendar day, 1 January 2005 – 31 December 2025	0.16 MB
hms_county_day_2006_2025.csv.gz	3,969,604	one county × one UTC day with at least one plume over the county’s land (sparse)	15.6 MB (gzip-compressed; about 111 MB as plain text)
hms_county_month_2006_2025.csv	745,920	one county × one month, 2006–2025: 3,108 counties × 240 months (complete)	21.7 MB

Clean variables

hms_smoke_plumes_2005_2025.csv: one row per plume; the key is plume_id. Read it with col_types as in chunk m02-read-back. Left to guess, read_csv() reads density as lgl, because the first 1,000 rows (2005) have no label, and every label becomes NA.

Variable	Type	Units	Description
plume_id	chr	—	Year and row number of the polygon in that year’s raw shapefile, e.g., 2023_00001. Row numbers count the raw file, so the 90 dropped polygons leave gaps
date	date	—	UTC date of the plume’s start time (the project’s day); 2005-08-05 to 2025-12-31
start_utc	datetime	UTC	Time of the first image used (raw Start)
end_utc	datetime	UTC	Time of the last image used (raw End); never before start_utc, at most 23 hours after it
satellite	chr	—	GOES, GOES-EAST, GOES-WEST, NOAA 20, NOAA 21, or Suomi NPP (raw, spaces trimmed)
density	chr	—	Light, Medium, or Heavy; empty (NA) when the analyst gave no label. Every plume from 2 June 2010 on has a label
area_km2	dbl	km²	Area of the repaired outline in CONUS Albers (EPSG:5070), an equal-area projection; median 636 km², up to 52.3 million km². Rounded to 0.1, so 8 tiny plumes show 0
centroid_lon	dbl	degrees east	Longitude of the outline’s center (computed in EPSG:5070, then converted back); negative = west
centroid_lat	dbl	degrees north	Latitude of the outline’s center

hms_calendar_2005_2025.csv: one row per UTC calendar day; the key is date. read_csv() reads n_plumes as dbl.

Variable	Type	Units	Description
date	date	—	Calendar day, 2005-01-01 to 2025-12-31
n_plumes	dbl	count	Plume polygons with that start date (after the 90 empty outlines are dropped); 0 on days with none
analysed	lgl	—	TRUE if at least one plume is dated that day. FALSE is ambiguous: either the analysts saw no smoke, or nobody analysed the day. The files do not say which
share_unlabeled	dbl	share (0–1)	Share of that day’s plumes without a density label; empty (NA, not applicable) on days with no plume

hms_county_day_2006_2025.csv.gz: one row per county and UTC day with at least one plume over the county’s land; the key is GEOID + date. Rows are sorted by date, then GEOID. The script writes the file one year at a time, appending each year to the end; read_csv() reads it as one file. A county-day that is absent means no plume overlapped the county that day. That is “no smoke seen” only if the calendar says the day was analysed. read_csv() reads the counts as dbl.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (2-digit state + 3-digit county), 2020 boundaries; joins to counties_2020.csv
date	date	—	UTC date of the plumes’ start time, 2006-01-01 to 2025-12-31
n_plumes	dbl	count	Number of plume polygons that touch the county’s land that day (1 to 46). Outlines drawn from GOES-East and GOES-West images, or at different times of day, can cover the same smoke, so this is not a count of separate smoke events
max_density	dbl	code	Densest plume over the county that day: 0 = no label, 1 = Light, 2 = Medium, 3 = Heavy
pop_share_any	dbl	share (0–1)	Share of the county’s 2020 population whose block-group center lies inside any plume that day. Can be 0 when a plume touches only the edge of the county
pop_share_medium_plus	dbl	share (0–1)	Same, for plumes labeled Medium or Heavy
pop_share_heavy	dbl	share (0–1)	Same, for plumes labeled Heavy

hms_county_month_2006_2025.csv: one row per county and month; the key is GEOID + year + month. Every county-month is present, and a 0 is a real zero: no plume touched the county on any analysed day of the month. read_csv() reads every column except GEOID as dbl.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code
year	dbl	—	Calendar year, 2006–2025
month	dbl	—	Month, 1–12
plume_days	dbl	days	Days with at least one plume over the county (rows in the county-day file)
medium_plus_days	dbl	days	Days whose densest plume is Medium or Heavy (max_density >= 2)
heavy_days	dbl	days	Days whose densest plume is Heavy (max_density == 3)
pop_weighted_plume_days	dbl	days	Sum of pop_share_any over the month: plume days weighted by the share of residents under the plume
analysed_days	dbl	days	Days in the month with at least one HMS plume anywhere (from the calendar); the same for every county
n_days	dbl	days	Days in the month (28–31)

Checks and cautions

• Smoke overhead is not smoke at breathing height. Analysts draw plumes from satellite pictures of the whole air column. Liu et al. (2024) compared HMS with airport visibility reports, air-quality monitors, and model estimates of near-surface smoke for 2010–2021: outside the West, surface smoke PM2.5 inside plumes is low and barely differs across the Light, Medium, and Heavy classes. Treat an eastern plume day as “smoke was overhead”, and use a surface measure (AQS monitors, Childs, CSU) for what people breathed. Reference: Liu, T., Panday, F. M., Caine, M. C., Kelp, M., Pendergrass, D. C., Mickley, L. J., Ellicott, E. A., Marlier, M. E., Ahmadov, R., and James, E. P. (2024), “Is the smoke aloft? Caveats regarding the use of the Hazard Mapping System (HMS) smoke product as a proxy for surface smoke presence across the United States,” International Journal of Wildland Fire 33(10), WF23148, doi:10.1071/WF23148.
• Agreement with the research project’s build (05c_hms.R, which uses data.table and county outlines simplified to about 200 m): the county-day file matches 3,969,409 of the project’s 3,969,528 county-days for 2006–2025. 195 county-days appear only here and 119 only in the project; all of them have one plume that grazes the county edge and covers no population center, so they come from the outline simplification. On matched rows, n_plumes agrees on 99.98% (662 differ) and max_density on 99.997% (117 differ), and the three population shares differ by at most 0.00005 (rounding). These differences have two sources. A few come from the 13 flat outlines the project keeps: repair turns them into lines, which can touch counties without covering any area. Only 19 of the 662 n_plumes differences fall on the 10 dates with such lines, and none of the max_density differences do. The rest come from the simplified county edges: a plume that just grazes a county can touch the exact outline but miss the simplified one, or the other way round. On the remaining 643 county-days this file has one plume more (435) or one fewer (208). In the county-month file, plume_days differs in 314 of 745,920 cells (by at most 1 day), and analysed_days and n_days agree everywhere. The calendar’s analysed flag is identical to the project’s; the project counts 13 more polygons on 10 days because it keeps 13 flat outlines that this script drops.
• Raw versus clean plumes. The 21 raw files hold 518,487 polygons; the plume table keeps 518,397. The 90 dropped are 77 single points or there-and-back lines and 13 flat outlines; none has an area.
• When density labels begin. No plume has a label in 2005 or 2006. The first labeled plume is dated 26 April 2007. Labels stay patchy: 19,614 of 19,872 raw plumes are unlabeled in 2007, 5,076 of 23,277 in 2008, 23,325 of 23,473 in 2009, and 7,437 of 27,230 in 2010. The last unlabeled plume is dated 1 June 2010; from 2 June 2010 every plume is labeled. Before then, max_density == 0 and a medium_plus_days of 0 do not mean the smoke was thin, only that nobody labeled it. Density-based measures are comparable over time only from mid-2010. A published block-group smoke series built from HMS also starts in June 2010: Vargo, J. A. (2020), “Time series of potential US wildland fire smoke exposures,” Frontiers in Public Health 8:126, doi:10.3389/fpubh.2020.00126.
• A day with no plume is ambiguous. The calendar has 61 days without any plume between 2006 and 2025 (for example 25 December in several years). On those days either the analysts saw no smoke or the day was not analysed, and the files cannot tell these apart. For example, 12 May 2023, during the Alberta fires, has no plume at all; that day was almost certainly not analysed, so treat it as missing, not as smoke-free. HMS starts on 5 August 2005, so the 216 days before that have no plume because there is no analysis; three days after 5 August 2005 also have none. In the county-month file, 42 months have analysed_days < n_days; divide by analysed_days, not n_days, if you want a share of days.
• What a missing county-day means. The county-day file is sparse: a county and day that is not in the file had no plume over its land. On an analysed day that is a real zero (“no smoke seen”); on a day that is not analysed it is unknown. Before joining it to a complete panel, fill n_plumes with 0 and the shares with 0 only for analysed days.
• A plume day can have a population share of 0. 278,631 of county-days (7.0%) have pop_share_any == 0: the plume touches the county’s land but misses every block-group center. In 79% of county-days the whole population is under a plume.
• More plumes in recent years. County-days per year rise from about 70,000–200,000 in 2006–2020 to 320,000–530,000 in 2021–2025 (the peak, 532,963, is 2023). Much of this is more smoke (2021 and 2023 were big fire years in Canada and the West), but the images and the analysts’ practice also changed: GOES-16 replaced the older GOES-East in December 2017, and GOES-19 replaced GOES-16 on 7 April 2025. Compare years with care, and use year fixed effects in regressions.
• Satellite labels change in October 2016. Until 31 October 2016 every plume says GOES; from 20 October 2016 plumes say GOES-EAST or GOES-WEST. Do not read the switch as a change in coverage.
• Plume size varies hugely. Areas run from under 1 km² (13 plumes) to 52.3 million km², with a median of 636 km². 1,062 plumes are larger than 10 million km²; the largest are Light plumes from the 2023 and 2025 Canadian seasons that stretch far beyond North America. For such huge outlines the area is only approximate, because the edges are drawn as straight lines between corners. Because one plume can cover a whole region, plume counts say more about how analysts split smoke into outlines than about how much smoke there was.
• Reading the files. Read the plume table with col_types (chunk m02-read-back); otherwise density is read as all NA. Keep GEOID as text. Reading the whole county-day file with read_csv() needs about 1 GB of memory, so do it in a fresh R session (the comment in chunk m02-read-county-day shows the call). The file is written in yearly pieces; this is invisible to read_csv(), gzip, and Python’s pandas.
• Run time and memory. On the instructor’s Mac (with other jobs running) the full script took about 15 minutes (883 and 884 seconds in two clean runs). To keep memory low it works in 5-day windows inside each year, tests population centers one day at a time, and writes the county-day file one year at a time instead of holding all 4 million rows. In those two runs the R process used at most 1.7 to 1.8 GB of memory (maximum resident set size); the “Memory” figure in Activity Monitor peaked at 0.8 to 1.0 GB. Plan on about 2 GB of free memory, and close other programs before you run the loop.

CSU kriged smoke PM2.5

This product estimates how much of the fine-particle pollution (PM2.5: particles 2.5 micrometers across or smaller) in the air near the ground came from smoke, every day from 2006 to 2025, on a grid of cells about 15 km wide over the contiguous U.S. A team of atmospheric scientists based mainly at Colorado State University (CSU) makes it. They start from the 24-hour averages measured at EPA ground monitors and fill in the map between monitors by kriging, a statistical method that predicts each cell from nearby monitors and gives more weight to monitors that are closer. They then use NOAA’s Hazard Mapping System (HMS), in which analysts trace smoke plumes on satellite images every day: on cell-days under a plume, smoke PM2.5 is total PM2.5 minus the cell’s normal non-smoke level (its seasonal “background”); on all other cell-days it is zero.

It is the research project’s main measure of the smoke that people breathe: one consistent method for every year, every county, and every day. It cannot tell you where the smoke came from (Canadian, western U.S., or agricultural fires), because it uses no fire locations and no winds. Between monitors it is an interpolation, so values in rural areas far from monitors are smooth guesses, and an HMS plume can sit high above clean surface air, so smoke PM2.5 on some plume days is only noise around zero.

Provider	Jeffrey Pierce, Bonne Ford, Katelyn O’Dell, Jennifer McGinnis, and Emily Fischer, published on Dryad (version 8, 2026-08-06). Dryad lists Pierce, McGinnis, and Fischer at Colorado State University, Ford at the Cooperative Institute for Research in the Atmosphere (CIRA, a CSU institute), and O’Dell at the U.S. Department of Energy. Inputs: EPA Air Quality System (AQS) monitors and NOAA HMS smoke plumes
Coverage	2006–2025, daily · a 309 × 189 grid over the contiguous U.S. that also covers nearby ocean, southern Canada, and northern Mexico. In the bundle: the raw grid for 2023 only (median background), and county files for all of 2006–2025 (the 3,108 contiguous-U.S. counties, DC included)
Raw unit	one grid cell (about 15 km wide) × day, in each of three daily arrays
Clean unit	one row = a county × day (2023 sample here; 2006–2025 in the instructor export) or a county × month (instructor export)
Licence and citation	CC0 1.0 (public-domain dedication: no restrictions, but cite the work). Data: Pierce, J., Ford, B., O’Dell, K., McGinnis, J., & Fischer, E. (2026). Spatially interpolated non-smoke and smoke PM2.5 concentrations for the US from 2006-2025 (Version 8) [Dataset]. Dryad. https://doi.org/10.5061/dryad.k0p2ngfhv. Method: O’Dell, K., Ford, B., Fischer, E. V., & Pierce, J. R. (2019). Contribution of wildland-fire smoke to US PM2.5 and its influence on recent trends. Environmental Science & Technology, 53(4), 1797–1804. https://doi.org/10.1021/acs.est.8b05430
Documentation	Dryad dataset page and README (a copy of the README is in raw/csu/README.md); O’Dell et al. (2019); processing code (Zenodo); NOAA HMS fire and smoke product

Raw files

The full product is 40 NetCDF files (20 years × two background definitions, mean and median), 514.6 to 516.0 MB each and 20.6 GB in all. The bundle ships one of them: 2023 with the median background. The county files for all years come from the instructor export (see the appendix).

File (in wildfire_data/raw/)	Size	Format	One row is
csu/krigedPM25_2023_v2_statfix_medianbk.nc	514.6 MB	NetCDF-4 (a file of labeled multi-dimensional arrays)	not rows but arrays: one value per grid cell × day for 309 × 189 cells × 365 days (21,316,365 cell-days) of 2023, median background
csu/README.md	0.005 MB	Markdown text	the provider’s README for Dryad version 8 (methods in brief, variable list)

Raw variables

The table covers csu/krigedPM25_2023_v2_statfix_medianbk.nc. The other yearly files use the same grid and the same main variables (the research project checked all 40). The file has eight dimensions: lony (309), lonx (189), date (365 days), welony (310), welonx (189), nslony (309), nslonx (190), and nsites (2,228 monitor sites). nsites is the file’s unlimited (record) dimension; the others have a fixed length. Dimensions are listed in R’s order, as ncdf4 shows them; tools that use C order, such as ncdump, show them reversed, e.g., PM25(date, lonx, lony). Despite its name, lony runs roughly west to east and lonx south to north. Every variable is stored as a double except doy (integer). All except doy declare 1e30 as the missing-value code; no value in the 2023 file uses it. The file’s only global attribute, description, reads “Krigged PM2.5 concentrations from EPA AQS surface monitors for Jan-Dec 2023created: 2025-01-13” (sic). Each variable also carries units, long_name, and var_desc attributes with a short label (for example, HMS_Smoke has the var_desc “Binary HMS Smoke: 1 = smoke, 0 = no smoke”). Read them with ncatt_get(nc, "PM25"); the descriptions below agree with them.

Variable	Dimensions	Units	Description
doy	date	stored as “date”; really a day count	Day of year of each 24-hour average, counted from 0: 0 = January 1, 364 = December 31
lon	lony, lonx	degrees	Longitude of each cell’s center (negative = west of Greenwich), from −134.2 to −65.8. It changes along both dimensions because the grid is curvilinear (laid out on a projected map, not along lines of longitude and latitude)
lat	lony, lonx	degrees	Latitude of each cell’s center, from 22.5 to 52.6
we_lon	welony, welonx	degrees	Longitude of the cell borders in the west–east direction: 310 points per row of 309 cells, one on each border between neighbors plus the two outer edges
we_lat	welony, welonx	degrees	Latitude of the same west–east border points
ns_lon	nslony, nslonx	degrees	Longitude of the cell borders in the north–south direction: 190 points per column of 189 cells
ns_lat	nslony, nslonx	degrees	Latitude of the same north–south border points
PM25	lony, lonx, date	ug/m3 (µg/m³)	24-hour average total PM2.5 concentration in micrograms per cubic meter of air (µg/m³), kriged from EPA AQS monitors. Kriging can undershoot below zero: 0.21% of 2023 cell-days are negative (minimum −26.9 µg/m³)
Background_PM25	lony, lonx, date	ug/m3 (µg/m³)	Seasonal non-smoke background: the median (in this medianbk file; the mean in meanbk files) of PM25 over the cell’s days in the season with no HMS smoke polygon. Seasons are quarters: January–March, April–June, July–September, October–December. At least 15 non-smoke days are required; if a season has fewer, days just before and after it are added. One value per cell and quarter, repeated on every day (checked in 2023: no cell changes within a quarter)
HMS_Smoke	lony, lonx, date	binary	1 = an HMS smoke polygon (light, medium, or heavy density) lies over the cell that day; 0 = none. Whether the cell’s center or any part of the cell must be covered is not documented by the provider. A plume can be anywhere in the air column, not only at the surface. 33.8% of 2023 cell-days in the whole rectangle are 1
testing_sites_longitudes	nsites	degrees	Longitude of each EPA AQS monitor site used to test the kriging by leave-one-out cross-validation (LOOCV): each monitor is left out in turn, the map is re-estimated without it, and the prediction is compared with what the monitor measured. The README calls it testing_sites_longitude
testing_sites_latitudes	nsites	degrees	Latitude of the same monitor sites (the README calls it testing_sites_latitude)
r_squared	nsites	unitless	R² between LOOCV predictions and the site’s measurements (2023 median across sites: 0.88)
mean_bias	nsites	ug m-3 (µg/m³)	Mean bias of the LOOCV predictions at the site; the sign convention (predicted minus measured, or the reverse) is not documented by the provider
mean_absolute_error	nsites	ug m-3 (µg/m³)	Mean absolute error of the LOOCV predictions at the site (2023 median: 1.48 µg/m³)
slope	nsites	unitless	Slope of a linear regression of LOOCV predictions and measurements at the site; which variable is on which side is not documented by the provider. Missing for 1 site in 2023
nobs	nsites	unitless	Number of daily monitor observations at the site available for LOOCV (1 to 365 in 2023)

Cleaning steps

The script R/m07_csu.R makes these decisions:

1. One year, a month at a time. The sample file holds 21 million cell-days in each of three arrays. The script reads one month at a time (about 1.8 million cell-days), so it needs about 1.5 to 1.7 GB of memory at most.
2. Dates from doy. doy starts at 0, so the date is January 1 plus doy.
3. Smoke = HMS flag × (total − background). The file has no smoke variable; we compute it for every cell-day. Cell-days without a plume get exactly 0. Small negative values are kept, as the authors advise: they are noise around zero, and dropping or flooring them would push average smoke up.
4. Population weights. People, not land, breathe smoke. Each of the 2020 block groups with residents goes to the grid cell whose center is nearest to its center of population, with distances measured in CONUS Albers (EPSG:5070). CONUS Albers is a flat map projection of the contiguous U.S. (“CONUS”) whose coordinates are in meters, so distances can be measured directly; module m00 explains CRS and EPSG codes. Block groups more than 25 km from every cell center are dropped; this only happens in Washington County, Maine, where the grid ends. A county’s weight on a cell is the share of the county’s people who live in it. The research project uses the same rule.
5. County-day values are population-weighted means of the cell values: smoke, total PM2.5, background, and the HMS flag (which becomes the share of residents under a plume). The grid has no missing values, so no weights need to be adjusted for gaps.
6. Rounding. PM2.5 values are rounded to 2 decimals and hms_share to 3. The check that smoke is 0 whenever hms_share is 0 runs on the unrounded values, because rounding can break it (see Checks and cautions).
7. A complete panel. The output has every county on every day of 2023 and no missing values.

The first chunk opens the file and lists its variables with their dimensions and units.

# The bundle ships one year of the CSU product: 2023, median background.
# nc_open() reads only the file's header (names, dimensions, units), so it is
# instant even though the file is about 500 MB.
csu_file <- file.path(raw_dir, "csu", "krigedPM25_2023_v2_statfix_medianbk.nc")
nc <- nc_open(csu_file)

# One row per variable: its dimensions, in the order R uses, and its units.
csu_vars <- tibble(
  variable = names(nc$var),
  dims = map_chr(nc$var, \(v) str_c(map_chr(v$dim, "name"), collapse = ", ")),
  units = map_chr(nc$var, "units")
)
csu_vars

Next, the grid. Each cell gets a number, and its center’s longitude and latitude come from the lon and lat matrices. The provider describes the cells as about 15 km wide; measured in CONUS Albers, neighboring centers are 16 km apart.

# The grid is curvilinear: cells are about 16 km apart on a projected map, so
# every cell has its own longitude and latitude, stored in two 309 x 189
# matrices. Careful with the names: the first dimension ("lony", 309 steps)
# runs roughly west to east, and the second ("lonx", 189 steps) south to north.
lon <- ncvar_get(nc, "lon")
lat <- ncvar_get(nc, "lat")
dim(lon)

# as.vector() reads a matrix column by column: cell 1 is lon[1, 1], cell 2 is
# lon[2, 1], and so on. The PM2.5 arrays are flattened the same way below, so
# cell numbers always line up.
cells <- tibble(cell = seq_along(lon), lon = as.vector(lon), lat = as.vector(lat))

# The rectangle covers more than the U.S.: ocean, southern Canada, and
# northern Mexico are in it too.
stopifnot(nrow(cells) == 309 * 189, !anyDuplicated(cells$cell), !anyNA(cells))
summarize(cells, across(c(lon, lat), list(min = min, max = max)))

The dates come from doy, which starts at 0.

# doy ("day of year") starts at 0, not 1: day 0 is January 1. Adding it to
# January 1 gives the calendar date.
doy <- ncvar_get(nc, "doy")
dates <- as.Date("2023-01-01") + doy

stopifnot(length(dates) == 365, all(doy == 0:364),
          max(dates) == as.Date("2023-12-31"))

This function reads any run of consecutive days into a long tibble with one row per cell × day: cell, lon, lat, date, total PM2.5, background PM2.5, and the HMS flag. ncvar_get() returns a 3-D array, and as.vector() flattens it cell by cell within each day, which is why cell, lon, and lat repeat times and date repeats each.

# Read consecutive days for every cell into a tibble with one row per cell x
# day: the cell's number and center, the date, and the three daily values.
# `start` and `count` pick a slice of the 3-D array (lony, lonx, date): all
# cells (count -1 means "to the end") and only the days we ask for. One month
# is about 1.8 million rows; the whole year at once would be 21 million rows,
# more memory than we need. The function uses `cells` and `dates` from the
# chunks above.
read_csu_days <- function(nc, day_index) {
  start <- c(1, 1, min(day_index))
  count <- c(-1, -1, length(day_index))
  get_var <- function(v) as.vector(ncvar_get(nc, v, start = start, count = count))
  n_days <- length(day_index)
  tibble(
    cell = rep(cells$cell, times = n_days),
    lon = rep(cells$lon, times = n_days),
    lat = rep(cells$lat, times = n_days),
    date = rep(dates[day_index], each = nrow(cells)),
    total_pm25 = get_var("PM25"),
    background_pm25 = get_var("Background_PM25"),
    hms = get_var("HMS_Smoke")
  )
}

Before averaging to counties, look at one month of cell-days and compute smoke.

# Look at one month first: June 2023, when smoke from fires in Quebec reached
# the Northeast. Smoke PM2.5 is total PM2.5 minus the seasonal background, but
# only on cell-days that lie under an HMS smoke plume (hms = 1); on all other
# cell-days it is exactly 0.
june <- read_csu_days(nc, which(month(dates) == 6)) |>
  mutate(smoke_pm25 = hms * (total_pm25 - background_pm25))

# The grid has no missing values, and hms is always 0 or 1.
stopifnot(nrow(june) == nrow(cells) * 30, !anyNA(june), all(june$hms %in% c(0, 1)))

# Under a plume, total PM2.5 can be below the background, which gives a small
# negative smoke value. The authors advise keeping these values: they are
# noise around zero, and dropping them would bias smoke upward.
june |>
  filter(hms == 1) |>
  summarize(cell_days = n(), mean_smoke = mean(smoke_pm25),
            share_negative = mean(smoke_pm25 < 0), max_smoke = max(smoke_pm25))

A map makes the grid concrete. It also shows where the grid ends: its eastern edge cuts through eastern Maine.

# Smoke PM2.5 on June 7, 2023, the day New York City's sky turned orange. Each
# square is one grid cell; the scale stops at 100 µg/m³ so that moderate
# smoke stays visible (darker cells can be far above 100).
states <- read_sf(file.path(clean_dir, "us_states_2020.gpkg")) |>
  filter(contiguous)

june |>
  filter(date == as.Date("2023-06-07")) |>
  ggplot() +
  geom_point(aes(lon, lat, color = smoke_pm25), shape = 15, size = 0.5) +
  geom_sf(data = states, fill = NA, color = "grey30", linewidth = 0.2) +
  scale_color_viridis_c(option = "inferno", direction = -1, limits = c(0, 100),
                        oob = scales::squish, name = "Smoke PM2.5\n(µg/m³)") +
  coord_sf(xlim = c(-125, -66.5), ylim = c(24.5, 49.5)) +
  labs(x = NULL, y = NULL)

Now the population weights. This chunk finds the nearest grid cell for every populated block group and measures the distance.

# Population weights. People, not land, breathe the smoke, so a county's value
# should average the cells where its people live. Each block group (from
# module m00) goes to the grid cell whose center is nearest to the block
# group's center of population. Block groups with no residents get no weight.
bg_centers <- read_csv(
  file.path(clean_dir, "census_bg_centers_2020.csv"),
  col_types = cols(bg_geoid = col_character(), GEOID = col_character(),
                   pop_2020 = col_double(), lat = col_double(), lon = col_double())
) |>
  filter(pop_2020 > 0)

# Distances need a projected CRS, one with coordinates in meters (see m00 for
# CRS and EPSG codes). We use CONUS Albers (EPSG:5070): a flat map of the
# contiguous U.S. ("CONUS") on which distances can be measured directly.
bg_points <- bg_centers |>
  st_as_sf(coords = c("lon", "lat"), crs = 4326) |>
  st_transform(5070)
cell_points <- cells |>
  st_as_sf(coords = c("lon", "lat"), crs = 4326) |>
  st_transform(5070)

nearest <- st_nearest_feature(bg_points, cell_points)
bg_cells <- bg_centers |>
  mutate(
    cell = cells$cell[nearest],
    dist_km = as.numeric(st_distance(bg_points, cell_points[nearest, ],
                                     by_element = TRUE)) / 1000
  )

# Inside the grid, no point is more than about 11 km (half the diagonal of a
# cell) from its nearest center.
summary(bg_cells$dist_km)

Block groups more than 25 km from a cell center are dropped, and the rest become county weights.

# The grid ends at about 67 degrees west, so the eastern edge of Washington
# County, Maine (GEOID 23029), lies outside it. A cell center more than 25 km
# away does not describe the air there, so those block groups are dropped
# (the research project uses the same 25 km rule). The county keeps the
# weight of its other block groups.
bg_cells |>
  filter(dist_km > 25) |>
  summarize(block_groups = n(), pop_dropped = sum(pop_2020), .by = GEOID)

# A county's weight on a cell = share of the county's people who live in it.
county_weights <- bg_cells |>
  filter(dist_km <= 25) |>
  summarize(pop = sum(pop_2020), .by = c(GEOID, cell)) |>
  mutate(weight = pop / sum(pop), .by = GEOID)

# Grain: one row per county x cell. Every one of the 3,108 counties has
# weights, and its weights add up to 1.
weight_sums <- county_weights |> summarize(total = sum(weight), .by = GEOID)
stopifnot(
  nrow(weight_sums) == 3108, all(abs(weight_sums$total - 1) < 1e-9),
  !anyDuplicated(select(county_weights, GEOID, cell))
)
# Number of cells per county: from 1 (small counties) to dozens (big ones).
count(county_weights, GEOID, name = "cells") |> pull(cells) |> summary()

The main step: for each month, compute smoke for every cell-day, join the weights, and take population-weighted means by county and day. It takes about a minute.

# Average the grid to counties one month at a time to keep memory low. The
# join is many-to-many on purpose: a cell appears once per day, and one cell
# can serve several counties. The county value is the population-weighted
# mean over the county's cells. Note that county smoke averages the cell
# smoke values; it is not hms_share x (total - background).
county_days_for_month <- function(m) {
  read_csu_days(nc, which(month(dates) == m)) |>
    mutate(smoke_pm25 = hms * (total_pm25 - background_pm25)) |>
    inner_join(county_weights, by = "cell", relationship = "many-to-many") |>
    summarize(
      smoke_pm25 = weighted.mean(smoke_pm25, weight),
      total_pm25 = weighted.mean(total_pm25, weight),
      background_pm25 = weighted.mean(background_pm25, weight),
      hms_share = weighted.mean(hms, weight),
      .by = c(GEOID, date)
    )
}

csu_county_day <- map(1:12, county_days_for_month) |> list_rbind()
nc_close(nc)

Round, check the grain, and write the file.

# Read only the columns we need, all as character: FIPS codes keep their
# leading zeros only when read as text.
counties <- read_csv(file.path(clean_dir, "counties_2020.csv"),
                     col_types = cols_only(GEOID = col_character(),
                                           county_name = col_character(),
                                           state_abbr = col_character()))

csu_sample <- csu_county_day |>
  transmute(
    GEOID, date,
    smoke_pm25 = round(smoke_pm25, 2),
    total_pm25 = round(total_pm25, 2),
    background_pm25 = round(background_pm25, 2),
    hms_share = round(hms_share, 3)
  ) |>
  arrange(GEOID, date)

# Grain: one row per county x day, 3,108 counties x 365 days, with no gaps and
# no missing values; hms_share is a share between 0 and 1. In 2023 the
# background is always positive (not in every year: see the cautions).
# Smoke is exactly 0 when no resident is under a plume. We check this on the
# unrounded values: after rounding, a tiny hms_share (below 0.0005) shows as 0
# while smoke can round to 0.01.
stopifnot(
  nrow(csu_sample) == 3108 * 365,
  !anyDuplicated(select(csu_sample, GEOID, date)),
  setequal(csu_sample$GEOID, counties$GEOID),
  setequal(csu_sample$date, dates),
  !anyNA(csu_sample),
  all(csu_county_day$smoke_pm25[csu_county_day$hms_share == 0] == 0),
  all(csu_sample$background_pm25 > 0),
  all(between(csu_sample$hms_share, 0, 1))
)

write_csv(csu_sample, file.path(clean_dir, "csu_sample_county_day_2023.csv.gz"), na = "")

Finally, a first look at the result. The last table shows why a zero in smoke_pm25 is not the same as a day without a plume.

# The five county-days with the most smoke in 2023, with county names.
csu_sample |>
  slice_max(smoke_pm25, n = 5) |>
  left_join(select(counties, GEOID, county_name, state_abbr), by = "GEOID")

# How common are smoke days, and how often is smoke slightly negative?
csu_sample |>
  summarize(
    mean_smoke = mean(smoke_pm25),
    # smoke day = more than half of residents under an HMS plume
    # (hms_share > 0.5), as in the project
    share_smoke_days = mean(hms_share > 0.5),
    share_negative = mean(smoke_pm25 < 0),
    min_smoke = min(smoke_pm25)
  )

# smoke_pm25 = 0 does not always mean "no plume". On some plume days the
# value rounds to 0.00 for one of two reasons: (a) the plume covers the
# county, but its total PM2.5 is about equal to its background; or (b) only
# a tiny share of residents is under the plume, so averaging in the zeros
# of everyone else pulls the county value to about 0. To find days without
# a plume, use hms_share, not smoke_pm25 == 0.
csu_sample |>
  filter(smoke_pm25 == 0) |>
  count(plume = case_when(hms_share == 0 ~ "none (hms_share = 0)",
                          hms_share == 1 ~ "whole county",
                          .default = "part of the county"))

Clean files

The first file is written by the code above. The other two come from the instructor export described in the appendix; they use the same method for every year from 2006 to 2025. All three use the median background.

File (in wildfire_data/clean/)	Rows	One row is	Size
csu_sample_county_day_2023.csv.gz	1,134,420	a county × day in 2023 (3,108 × 365), built here from the raw 2023 file	7.08 MB (36.9 MB uncompressed)
csu_smoke_county_day/csu_smoke_county_day_<year>.csv.gz, 20 files for 2006–2025	1,134,420 per year (1,137,528 in leap years); 22,703,940 in all	a county × day (instructor export)	5.9–7.1 MB each (about 37 MB uncompressed); 124.8 MB in all
csu_smoke_county_month_2006_2025.csv	745,920	a county × month, January 2006 to December 2025 (3,108 × 240; instructor export)	41.07 MB

Clean variables

csu_sample_county_day_2023.csv.gz and the yearly csu_smoke_county_day_<year>.csv.gz files share these columns. Rows are sorted by GEOID and date, and no cell is missing. Read a file with read_csv() directly; it unzips .gz files on the fly.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (state + county), 2020 boundaries; joins to counties_2020.csv
date	date	—	Day of the 24-hour average (from the raw doy)
smoke_pm25	dbl	µg/m³	Smoke PM2.5: population-weighted mean over the county’s cells of HMS_Smoke × (PM25 − Background_PM25). Exactly 0 when no resident lives under a plume (hms_share = 0): a true zero, not a missing value. The reverse does not hold: it can also be 0.00 on plume days (1,103 county-days in 2023). Because hms_share is rounded to 3 decimals, 7 county-days in 2006–2025 show hms_share = 0 with smoke_pm25 = 0.01. Can be negative; keep those values. 2 decimals
total_pm25	dbl	µg/m³	Total PM2.5: population-weighted mean of the kriged PM25, smoke and non-smoke together. Rarely negative, where kriging undershoots (516 county-days in 2023; 217 to 2,478 per year). 2 decimals
background_pm25	dbl	µg/m³	Non-smoke background: population-weighted mean of the seasonal median Background_PM25. Constant within each quarter, so it changes only on January 1, April 1, July 1, and October 1. Positive except in July–September 2016 in 11 counties of North Dakota and Minnesota (minimum −0.69) and July–September 2022 in Douglas County, Wisconsin (−0.35): kriged PM2.5 can be negative, and so can its median. 2 decimals
hms_share	dbl	share, 0 to 1	Share of the county’s 2020 population living in grid cells under an HMS smoke plume that day (population-weighted mean of HMS_Smoke). 3 decimals, so a share below 0.0005 shows as 0 (31 to 70 county-days per year). The project calls a county-day a smoke day when hms_share > 0.5

csu_smoke_county_month_2006_2025.csv: one row per county × month, sorted by GEOID, year, and month, no missing cells. Monthly values were computed by the research project from unrounded daily values, then rounded. read_csv() reads year, month, and the counts as dbl and csu_release as a date.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code, 2020 boundaries
year	dbl	—	Calendar year, 2006 to 2025
month	dbl	—	Calendar month, 1 (January) to 12 (December)
smoke_pm25	dbl	µg/m³	Monthly mean of daily smoke PM2.5 (the project’s monthly smoke measure, S). 2 decimals
total_pm25_mean	dbl	µg/m³	Monthly mean of daily total PM2.5. 2 decimals
smoke_max	dbl	µg/m³	Highest daily smoke PM2.5 in the month. 2 decimals
hms_days	dbl	days	Days with hms_share > 0.5 (most residents under a plume)
days_gt5	dbl	days	Days with daily smoke PM2.5 > 5 µg/m³
days_gt10	dbl	days	Days with daily smoke PM2.5 > 10 µg/m³
days_gt15	dbl	days	Days with daily smoke PM2.5 > 15 µg/m³
days_gt35	dbl	days	Days with daily smoke PM2.5 > 35 µg/m³ (35 µg/m³ is the EPA 24-hour standard for total PM2.5; here only the smoke part is counted)
n_days_obs	dbl	days	Days in the month with a value; equals the number of days in the month in every row
complete_month	lgl	—	TRUE when every day of the month has a value; TRUE in every row
csu_release	date	—	Release: the date Dryad published the file behind the month: 2025-03-21 for 2006–2023, 2026-02-25 for 2024, 2026-08-06 for 2025. This is not the date CSU made the file: the NetCDF records its creation date in its description attribute (2025-01-13 for 2006–2023, 2025-10-22 for 2024, 2026-07-30 for 2025)

Checks and cautions

• The sample matches the research project. We compared csu_sample_county_day_2023.csv.gz with the project’s population-weighted, median-background county-day values for 2023 (built by its script 05g_csu.R). All 1,134,420 county-days are in both files. The correlation is 1.0000 for all four variables. Mean absolute differences are 0.0012 µg/m³ (smoke), 0.0025 µg/m³ (total), 0.0025 µg/m³ (background), and 0.000006 (hms_share), and the largest is 0.005, which is rounding. Every value equals the project’s value rounded the same way. The method is the same: nearest cell center in EPSG:5070, the 25 km cut, and population shares from the same 2020 block groups. The project also re-weights over non-missing cells, which changes nothing because the grid has no gaps. The sample is identical, cell by cell, to csu_smoke_county_day_2023.csv.gz from the instructor export.
• Negative smoke is normal; keep it. On a plume day, total PM2.5 can be below the background (for example, when the plume stays aloft and the air at the ground is cleaner than usual). In 2023, 19% of plume cell-days are negative (minimum −31.7 µg/m³), and after averaging to counties 6.8% of county-days are negative (minimum −18.18 µg/m³). The share of negative county-days runs from 1.4% (2006) to 7.9% (2024). Do not drop them or set them to zero: that would bias smoke upward.
• What zero means. When hms_share = 0, smoke_pm25 = 0. This is a true zero, not a missing value: no resident of the county lived under an HMS plume that day, as on 53.7% of 2023 county-days. The reverse does not hold. smoke_pm25 can round to 0.00 on plume days: 1,103 county-days in 2023, for two reasons. (a) The plume covers the county, but its total PM2.5 is about equal to its background: 453 of the 1,103 have hms_share = 1 (for example, Coosa County, Alabama, on 2023-05-30, where total and background are both 6.92 µg/m³). (b) Only a tiny share of residents is under the plume, so the zeros of everyone else pull the county average to about 0: of the other 650, 72% have hms_share below 0.05 (median 0.014). Every year from 2006 to 2025 has 354 to 1,219 such county-days. To find days without a plume, use hms_share, not smoke_pm25 == 0. Rounding also works the other way: a county with a tiny share of residents under a plume (below 0.0005) shows hms_share = 0, and on 7 such county-days in 2006–2025 (for example, San Diego County on 2007-11-01) smoke_pm25 is 0.01. When only part of the county is under a plume (2.3% of 2023 county-days have hms_share strictly between 0 and 1), smoke_pm25 averages the smoke of the covered cells with zeros for the rest. County smoke is therefore not hms_share × (total_pm25 − background_pm25).
• 2023 in numbers. Mean county-day smoke PM2.5 is 2.39 µg/m³, the highest of 2006–2025 (the yearly mean ranges from 0.11 in 2009 to 2.39 in 2023). 45% of county-days have hms_share > 0.5. The largest county-day value is 237.70 µg/m³ (Spokane County, Washington, on 2023-08-19); on 2023-06-07, Lehigh County, Pennsylvania, reached 229 µg/m³.
• Smoke from where? CSU smoke counts smoke from all fires: Canadian, western U.S., prescribed burns, and farm fires. It cannot separate them. The project links smoke to Canadian fires with fire detections and winds from other modules.
• Median or mean background. This bundle uses the median background, as the instructor chose. The provider recommends the mean background for most uses, and the research project’s current design uses the mean for its headline estimates and the median as a robustness check. Smoke is higher with the median background: across 2006–2025 the average county-month smoke is 0.60 µg/m³ with the median and 0.52 µg/m³ with the mean.
• Kriging smooths, and it comes from the monitors. In rural areas far from monitors, values are interpolated from distant monitors and will miss local smoke. In 2023, the provider’s leave-one-out test at 2,228 monitor sites had a median R² of 0.88 and a median mean absolute error of 1.48 µg/m³. The project compared county-day CSU total PM2.5 with the county’s EPA monitor readings on the 2,884,953 county-days with a monitor (2006–2025): the correlation is 0.967 (0.974 on smoke days, 0.982 in 2023). This is a check of the build, not an independent test, because CSU is made from those same monitors.
• Where the grid ends. The grid’s eastern edge runs through Washington County, Maine (GEOID 23029). 19 block groups there, with 17,089 of the county’s 31,095 residents (55%), are more than 25 km from every cell center and get no weight, so that county’s values describe its western part only. Elsewhere every populated block group is within 11.5 km of a cell center.
• What a day is. The provider does not say which clock defines a day. EPA’s daily monitor averages, which the kriging starts from, cover the local calendar day at each monitor. Keep this in mind when you match CSU days to data stamped in UTC (Coordinated Universal Time, the clock of most satellite products).
• Releases. Years 2006–2023 come from the CSU release of 2025-03-21, 2024 from the release of 2026-02-25, and 2025 from the release of 2026-08-06 (all in Dryad version 8). These are the dates Dryad published the files. CSU made the files earlier: the NetCDF description attribute gives 2025-01-13 for 2006–2023, 2025-10-22 for 2024, and 2026-07-30 for 2025. The 2006–2023 files were made in one batch; 2024 and 2025 were separate, later runs, and the provider does not say whether anything in the method changed between them. Be careful when you compare 2024 or 2025 with earlier years. A later release could also revise recent years.
• Monthly versus daily files. Monthly values were computed from unrounded daily values. If you rebuild them from the rounded daily files, means differ by at most 0.006 µg/m³, and hms_days can differ in rare cases (0.02% of county-months in our test of 2006, 2023, and 2025), when hms_share is within rounding of 0.5.
• Memory and time. R/m07_csu.R runs in about 1 to 1.5 minutes, depending on how busy the computer is, with a peak memory use of about 1.5 to 1.7 GB. Reading the whole year at once instead of a month at a time would need several GB. If you run the script from the Terminal with Rscript R/m07_csu.R instead of in RStudio, R saves the map to a file called Rplots.pdf in your project folder; you can delete it.

Childs et al. smoke PM2.5 (Stanford ECHO Lab)

This file estimates smoke PM2.5 for every county and day: the part of fine particulate matter (particles smaller than 2.5 micrometers, PM2.5) that comes from wildfire smoke, in micrograms per cubic meter (µg/m³). Marissa Childs, Marshall Burke, and colleagues at Stanford’s Environmental Change and Human Outcomes (ECHO) Lab built it. A 10 km grid cell has a smoke day when a smoke plume drawn by NOAA’s Hazard Mapping System (HMS) from satellite images touches it. When clouds may hide plumes, it also has one if most of its satellite aerosol readings are missing and many simulated air trajectories from fires (from NOAA’s HYSPLIT model) pass low overhead. At EPA air monitors, smoke PM2.5 on a smoke day is the PM2.5 above the monitor’s usual level on non-smoke days; a machine-learning model (gradient-boosted trees) trained on satellite, weather, and fire data predicts that amount for every smoke-day grid cell, and the authors average the cells to counties, weighting by population and area. The method described here is from the 2022 paper (v1.0); v2.0 beta extends it to 2023 and may differ in details.

For this project it is the main daily measure of smoke exposure in U.S. counties from 2006 to 2023, built the same way for every county, so you can count smoke days and compare smoke across places; compare years with care (see “Comparing years” under “Checks and cautions”). It cannot tell you where the smoke came from: Canadian and U.S. fires are pooled. It is a model estimate, not a measurement, it says nothing about non-smoke pollution, and it stops at the end of 2023.

Provider	Stanford Environmental Change and Human Outcomes (ECHO) Lab (Childs et al.). Version 2.0 beta, file dated 2024-09-04, downloaded on 2026-09-27 from the lab’s Dropbox folder
Coverage	1 January 2006 – 31 December 2023 · contiguous U.S., 3,108 counties (DC included)
Raw unit	one row = a county on a smoke day (non-smoke days are left out)
Clean unit	one row = a county-day, every day (daily files); a county-month (monthly file)
Licence and citation	The provider states no licence for the v2.0 beta files and calls them “preliminary and subject to change”; the v1.0 release on Harvard Dataverse is CC BY-SA 4.0. Cite both papers and name the version (v2.0 beta, file of 2024-09-04). For v2.0 beta, the ECHO Lab page names this source (in review): Childs, M., Martins, M., Wilson, A., Qiu, M., Heft-Neal, S., and Burke, M. “Growing wildfire-derived PM2.5 across the contiguous U.S. and implications for air quality regulation.” EarthArXiv preprint, doi:10.31223/X57H95 (version 1, 9 December 2024; version 2, 18 August 2025). For the method (v1.0): Childs, M. L., Li, J., Wen, J., Heft-Neal, S., Driscoll, A., Wang, S., Gould, C. F., Qiu, M., Burney, J., and Burke, M. (2022). “Daily Local-Level Estimates of Ambient Wildfire Smoke PM2.5 for the Contiguous US.” Environmental Science & Technology 56(19): 13607–13621, doi:10.1021/acs.est.2c02934
Documentation	ECHO Lab wildfire smoke page; README__v2.0beta_20221104.txt in the raw folder; Childs et al. v2.0 preprint (EarthArXiv); Childs et al. (2022), Methods section “Defining Smoke Days”; code repository and README; county aggregation script; v1.0 on Harvard Dataverse

Raw files

File (in wildfire_data/raw/)	Size	Format	One row is
childs/v2.0beta/smokePM2pt5_predictions_daily_county_20060101-20231231__v2.0beta_20240904.csv	120.5 MB	comma-separated text, ASCII, header row; GEOID and date in double quotes	a county on a smoke day (3,317,195 rows, 3,108 counties)
childs/v2.0beta/README__v2.0beta_20221104.txt	0.003 MB	plain text	the provider’s notes on the county file (no data)

The research project also holds version 1.0 (2006–2020, from Harvard Dataverse). It is not in the student bundle, because the two versions are different model runs and must not be mixed (see “Checks and cautions”).

Raw variables

The CSV file has three columns. The README file has no variables.

Variable	Type	Description
GEOID	text in double quotes, 5 characters	County identifier (GEOID): 2-digit state FIPS code + 3-digit county FIPS code, e.g., 36051 = Livingston County, New York. The README says it matches the GEOID column of the 2019 TIGER/Line county shapefile for the contiguous U.S.
date	text in double quotes, 8 digits (YYYYMMDD)	Day of the prediction, e.g., 20230607 = 7 June 2023. The README does not say whether days follow local time or UTC (Coordinated Universal Time, the clock that satellites and weather models use). The model is trained on EPA daily averages, which use local calendar days, and its smoke days come from HMS’s daily plume analysis. Treat date as a local calendar day. It may be off by one day from UTC-day products such as ERA5 or EMBER
smokePM_pred	decimal number	Predicted daily smoke PM2.5 in µg/m³ on a smoke day. A county-day is in the file if any 10 km cell overlapping the county has a smoke day. The county value is the average over all overlapping 10 km cells, weighted by population and area of overlap; cells without smoke count as 0 in that average. A value of 0 means the county-day was a smoke day but had no elevated PM2.5. Days that are not in the file are non-smoke days, and their smoke PM2.5 is 0 by construction

Cleaning steps

The script R/m05_childs.R makes these decisions:

1. Read the keys with the right types. GEOID is read as text so the leading zero of codes such as 01001 survives. The date is text like 20060125; without a format, readr would read it as the number 20060125.
2. Check the county lists. The raw file has exactly the 3,108 counties of counties_2020.csv: no extra county and no county without rows. The script still keeps only universe counties, so it would work if the lists differed.
3. Absence means zero. The raw file lists smoke days only. The README says a county-day that is not in the file is a non-smoke day with smoke PM2.5 = 0. complete() adds those days and fills them with 0, and smoke_day records which rows came from the raw file. If you average only the raw rows, you average smoke days only and overstate smoke exposure (six times over for 2006–2023).
4. One year at a time. A complete year is 3,108 × 365 ≈ 1.1 million rows. All 18 years at once would be 20.4 million rows, so the script builds and writes one year per step.
5. Monthly summaries from unrounded values. Monthly means include the zeros, so they divide by the number of days in the month. Day counts use strict thresholds (> 5, > 15, > 35 µg/m³). These echo the WHO annual guideline (5 µg/m³), the WHO 24-hour guideline (15 µg/m³), and the U.S. EPA 24-hour standard (35 µg/m³) for total PM2.5; here they apply to the smoke part alone.
6. Round for storage. PM2.5 values are stored with 2 decimals. Each daily file is about 29 MB as plain text, so it is gzipped (.csv.gz); read_csv() reads it directly.

The first chunk reads the raw file with explicit column types and gives the value column a clearer name.

# The provider's file has one row per county per SMOKE day only. GEOID is
# quoted text in the file ("01001"), but we still declare it as character so
# the leading zero can never be lost. The date is stored as text such as
# "20060125" (YYYYMMDD). Left to guess, readr would read it as the number
# 20060125, so we give the format and get a real Date instead.
childs_file <- file.path(
  raw_dir, "childs", "v2.0beta",
  "smokePM2pt5_predictions_daily_county_20060101-20231231__v2.0beta_20240904.csv"
)

childs_raw <- read_csv(
  childs_file,
  col_types = cols(
    GEOID = col_character(),
    date = col_date(format = "%Y%m%d"),
    smokePM_pred = col_double()
  )
) |>
  rename(smoke_pm25 = smokePM_pred)

childs_raw

Before building on the file, check what the README promises: one row per county-day, dates inside 2006–2023, and no missing or negative values. The count by year shows that the file has many more smoke county-days after 2020 (see “Comparing years” under “Checks and cautions”).

# Check what the README promises before we build on it: one row per
# county-day, dates inside 2006-2023, no missing values (a date that failed
# to parse would show up as NA), and no negative smoke PM2.5.
stopifnot(
  !anyNA(childs_raw),
  all(str_detect(childs_raw$GEOID, "^[0-9]{5}$")),
  min(childs_raw$date) >= ymd("2006-01-01"),
  max(childs_raw$date) <= ymd("2023-12-31"),
  min(childs_raw$smoke_pm25) >= 0,
  nrow(distinct(childs_raw, GEOID, date)) == nrow(childs_raw)
)

# Smoke-day rows per year. The file gets much longer after 2020: many more
# county-days count as smoke days (2023 is the big Canadian smoke year).
childs_raw |>
  count(year = year(date), name = "smoke_county_days")

Next, compare the raw counties with the county universe in both directions. Both lists below are empty: the 3,108 counties match exactly.

# Our county universe is the 3,108 contiguous-U.S. counties plus DC in
# counties_2020.csv. Compare the two lists in both directions before joining.
counties <- read_csv(
  file.path(clean_dir, "counties_2020.csv"),
  col_types = cols(.default = col_character()),
  col_select = c(GEOID, county_name, state_abbr)
)

raw_geoids <- childs_raw |> distinct(GEOID)

# In the raw file but not in our universe: these rows would be dropped.
raw_geoids |> anti_join(counties, by = "GEOID")

# In our universe but never in the raw file: these counties would get 0 smoke
# on every day, so we want to know if there are any.
counties |> anti_join(raw_geoids, by = "GEOID")

# Keep only smoke days of counties in the universe.
childs_smoke <- childs_raw |> semi_join(counties, by = "GEOID")
nrow(childs_smoke)

This chunk shows why the missing days matter. In Livingston County, New York, 2023 had 162 smoke days. Their mean is 6.73 µg/m³, but the mean over all 365 days is 2.99 µg/m³. Over all counties and days of 2006–2023, the raw rows average 3.80 µg/m³ and the complete panel 0.62 µg/m³.

# KEY IDEA: a county-day that is missing from the raw file is NOT missing
# data. The README says those days had no smoke, so smoke PM2.5 = 0 there.
# A mean over the raw rows is a mean over smoke days only, and it overstates
# how much smoke people breathed over a month or a year.

# Example: Livingston County, NY (GEOID 36051, home of SUNY Geneseo), 2023.
livingston_2023 <- childs_smoke |>
  filter(GEOID == "36051", year(date) == 2023)

tibble(
  smoke_days = nrow(livingston_2023),
  mean_over_raw_rows = mean(livingston_2023$smoke_pm25),
  mean_over_all_365_days = sum(livingston_2023$smoke_pm25) / 365
)

# The same comparison for the whole file: 3,108 counties x 6,574 days.
all_dates <- seq(ymd("2006-01-01"), ymd("2023-12-31"), by = "day")
tibble(
  mean_over_raw_rows = mean(childs_smoke$smoke_pm25),
  mean_over_all_county_days = sum(childs_smoke$smoke_pm25) /
    (nrow(counties) * length(all_dates))
)

The next chunk defines a function that builds the complete panel for one year with complete().

# complete() adds every county-day that is missing from the raw file and
# fills it with 0 smoke PM2.5, as the README tells us to. smoke_day records
# whether the row came from the raw file (TRUE) or was added (FALSE).
# We work one year at a time: a year is 3,108 x 365 = 1.1 million rows,
# while all 18 years at once would be 20.4 million rows.
complete_year <- function(yr) {
  year_dates <- seq(make_date(yr, 1, 1), make_date(yr, 12, 31), by = "day")
  childs_smoke |>
    filter(year(date) == yr) |>
    mutate(smoke_day = TRUE) |>
    complete(
      GEOID = counties$GEOID,
      date = year_dates,
      fill = list(smoke_pm25 = 0, smoke_day = FALSE)
    ) |>
    arrange(GEOID, date)
}

A second function turns one year of the complete daily panel into county-month summaries.

# Monthly summaries of one year of the COMPLETE daily panel. The zeros are
# already in, so mean() divides by the number of days in the month, not by
# the number of smoke days. We use the unrounded daily values here.
summarize_months <- function(day_panel) {
  day_panel |>
    mutate(year = year(date), month = month(date)) |>
    summarize(
      smoke_mean = mean(smoke_pm25),
      smoke_max = max(smoke_pm25),
      smoke_days = sum(smoke_day),
      days_gt5 = sum(smoke_pm25 > 5),
      days_gt15 = sum(smoke_pm25 > 15),
      days_gt35 = sum(smoke_pm25 > 35),
      .by = c(GEOID, year, month)
    )
}

Try the function on 2023 and check it before running all years. In 2023, 47.8% of county-days were smoke days, and the mean smoke PM2.5 over all county-days was 2.45 µg/m³.

# Try the function on 2023, the year of the big Canadian fires.
day_2023 <- complete_year(2023)
day_2023

# Every county has all 365 days, the TRUE rows are exactly the raw rows,
# and every added row has 0 smoke.
stopifnot(
  nrow(day_2023) == nrow(counties) * 365,
  sum(day_2023$smoke_day) == sum(year(childs_smoke$date) == 2023),
  all(day_2023$smoke_pm25[!day_2023$smoke_day] == 0)
)

# Share of county-days with smoke in 2023, and the mean over all county-days.
day_2023 |>
  summarize(share_smoke_days = mean(smoke_day), mean_smoke_pm25 = mean(smoke_pm25))

The loop builds, checks, and writes one daily file per year with map(), and keeps each year’s monthly summary. It takes a minute or two.

# Build, check, and write one file per year. Each file is about 29 MB as
# plain text, so we gzip it (.csv.gz); read_csv() reads it directly.
# The loop also returns each year's monthly summary for the next chunk.
day_dir <- file.path(clean_dir, "childs_smoke_county_day")
dir.create(day_dir, recursive = TRUE, showWarnings = FALSE)

childs_month <- map(2006:2023, \(yr) {
  day_panel <- complete_year(yr)
  # Grain: one row per county x day, for every county and every day.
  stopifnot(
    nrow(day_panel) == nrow(counties) * yday(make_date(yr, 12, 31)),
    nrow(distinct(day_panel, GEOID, date)) == nrow(day_panel),
    !anyNA(day_panel),
    sum(day_panel$smoke_day) == sum(year(childs_smoke$date) == yr)
  )
  day_panel |>
    mutate(smoke_pm25 = round(smoke_pm25, 2)) |>
    write_csv(file.path(day_dir, str_glue("childs_smoke_county_day_{yr}.csv.gz")), na = "")
  message("wrote ", yr)
  summarize_months(day_panel)
}) |>
  list_rbind()

The last writing step rounds the monthly values, checks the grain and the logic of the day counts, and writes the monthly file.

# Round only now, after every summary was computed from unrounded values.
childs_month <- childs_month |>
  mutate(across(c(smoke_mean, smoke_max), \(x) round(x, 2))) |>
  arrange(GEOID, year, month)

# Grain: one row per county x month, 3,108 counties x 216 months, no gaps.
# Every raw smoke day is counted exactly once, and the day counts nest:
# days > 35 <= days > 15 <= days > 5 <= smoke days <= days in the month.
stopifnot(
  nrow(childs_month) == nrow(counties) * 18 * 12,
  nrow(distinct(childs_month, GEOID, year, month)) == nrow(childs_month),
  !anyNA(childs_month),
  sum(childs_month$smoke_days) == nrow(childs_smoke),
  all(childs_month$smoke_mean <= childs_month$smoke_max),
  all(childs_month$days_gt35 <= childs_month$days_gt15),
  all(childs_month$days_gt15 <= childs_month$days_gt5),
  all(childs_month$days_gt5 <= childs_month$smoke_days),
  all(childs_month$smoke_days <=
        days_in_month(make_date(childs_month$year, childs_month$month)))
)

write_csv(childs_month, file.path(clean_dir, "childs_smoke_county_month_2006_2023.csv"), na = "")

Finally, this chunk shows how to read the daily files back: pick the years you need and give the column types.

# How to read the daily files later: list them, keep the years you need,
# and give col_types so GEOID stays text and smoke_day stays TRUE/FALSE.
# All 18 years together are 20.4 million rows, so read only what you need.
day_files <- list.files(day_dir, pattern = "\\.csv\\.gz$", full.names = TRUE)

childs_day_2021_2023 <- day_files |>
  str_subset("_202[1-3]\\.csv\\.gz$") |>
  read_csv(col_types = cols(
    GEOID = col_character(),
    date = col_date(),
    smoke_pm25 = col_double(),
    smoke_day = col_logical()
  ))

stopifnot(
  length(day_files) == 18,
  nrow(childs_day_2021_2023) == nrow(counties) * (365 + 365 + 365),
  sum(childs_day_2021_2023$smoke_day) == sum(year(childs_smoke$date) >= 2021)
)

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
childs_smoke_county_day/childs_smoke_county_day_<year>.csv.gz (18 files, 2006–2023)	1,134,420 per year (1,137,528 in the leap years 2008, 2012, 2016, 2020); 20,431,992 in total	a county-day	3.0–4.3 MB each (about 29 MB unzipped); 61.0 MB in total
childs_smoke_county_month_2006_2023.csv	671,328 (3,108 counties × 216 months)	a county-month	19.5 MB

Clean variables

All 18 daily files share these columns. Rows are sorted by GEOID and date. Every county has every day of the year, and no cell is missing.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (state + county). The provider aggregated to 2019 TIGER/Line counties, and their codes are identical to the 3,108 GEOIDs in counties_2020.csv
date	date	—	Day of the prediction (YYYY-MM-DD), from the raw date. Treat it as a local calendar day; it may be off by one day from UTC-day products
smoke_pm25	dbl	µg/m³	Daily smoke PM2.5, 2 decimals. On a smoke day it is the raw smokePM_pred; on every other day it is 0, as the README instructs. It can be 0.00 on a smoke day
smoke_day	lgl	—	TRUE if the county-day is in the raw file (a smoke day for the provider); FALSE if the row was added by complete(). A county-day is in the raw file if any 10 km cell overlapping the county has a smoke day, so TRUE means smoke somewhere in the county, not everywhere in it. Cells without smoke count as 0 in the population-and-area weighted mean behind smoke_pm25

childs_smoke_county_month_2006_2023.csv: one row per county-month, sorted by GEOID, year, and month. Every county has all 216 months, and no cell is missing. All values come from the unrounded daily values.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (state + county). The provider aggregated to 2019 TIGER/Line counties, and their codes are identical to the 3,108 GEOIDs in counties_2020.csv
year	dbl	—	Calendar year, 2006–2023
month	dbl	—	Calendar month, 1–12
smoke_mean	dbl	µg/m³	Mean daily smoke PM2.5 over all days of the month, zeros on non-smoke days included (the sum over smoke days divided by the number of days in the month), 2 decimals
smoke_max	dbl	µg/m³	Highest daily smoke PM2.5 in the month, 2 decimals; 0 if the month has no smoke day
smoke_days	dbl	days	Number of smoke days (county-days in the raw file), 0–31
days_gt5	dbl	days	Number of days with smoke PM2.5 above 5 µg/m³
days_gt15	dbl	days	Number of days with smoke PM2.5 above 15 µg/m³
days_gt35	dbl	days	Number of days with smoke PM2.5 above 35 µg/m³

Checks and cautions

• Complete panel. The daily files hold 20,431,992 county-days (3,108 counties × 6,574 days), with no duplicate county-day and no missing cell. 3,317,195 of them (16.2%) are smoke days; all other rows have smoke_pm25 = 0. The monthly file has 671,328 rows: 3,108 counties × 216 months. 63.9% of county-months have at least one smoke day.
• Matches the project build. The smoke-day rows are the same 3,317,195 county-days as in the research project’s v2.0 beta county-day file, key for key, and the values differ only by rounding (at most 0.005 µg/m³). All 671,328 county-months match the project’s monthly file: smoke_mean and smoke_max agree within 0.005 µg/m³ (rounding), and smoke_days, days_gt5, days_gt15, and days_gt35 are identical.
• County lists match. The raw file has 3,108 GEOIDs, and they are exactly the 3,108 GEOIDs of counties_2020.csv. No raw row was dropped, and no county is all zeros because it was missing from the raw file.
• Absence means zero. The daily mean over all county-days of 2006–2023 is 0.62 µg/m³; the mean of the raw rows alone is 3.80 µg/m³, six times higher. In Livingston County, New York, smoke PM2.5 peaked at 92.58 µg/m³ on 7 June 2023, during the Canadian smoke episode.
• Use smoke_day to count smoke days. 137,690 smoke-day rows show smoke_pm25 = 0.00. Of these, 14,796 are exact zeros in the raw file (a smoke day without elevated PM2.5 at the ground, as the README puts it) and 122,894 are below 0.005 µg/m³ and round to 0. Partial coverage also produces near-zero values: if smoke touches only a few of the 10 km cells in a county, the other cells count as 0 in the county average. Counting smoke_pm25 > 0 would miss these rows.
• Smoke somewhere in the county. smoke_day is TRUE when at least one 10 km cell overlapping the county is a smoke day. Big counties have more cells, so they are smoke days a little more often. Within the same state, the largest quarter of counties (by land area) are smoke days on 17.0% of days and the smallest quarter on 15.6%.
• Rebuilding monthly values from the daily files. The monthly file uses unrounded daily values. If you rebuild it from the 2-decimal daily files, smoke_days and smoke_max come out the same, but days_gt5 is lower in 931 county-months (932 days in all), days_gt15 in 66, and days_gt35 in 5, because a value such as 5.004 is stored as 5.00. smoke_mean differs by 0.01 µg/m³ in 8,059 county-months (1.2%).
• Days with no smoke row anywhere. On 99 of the 6,574 days, no county has a row, so every county gets 0. 64 of these days fall in December or January, when smoke is rare. On 54 of them, the project’s copy of the NOAA HMS smoke archive has no plume polygons at all, so a 0 on those days may mean “no satellite smoke analysis” rather than clean air. The converse also happens: on 7 other days the HMS archive has no plumes but the file still has smoke rows (for example 2,714 of 3,108 counties on 12 May 2023). The model’s second route (fire trajectories when clouds hide plumes) can produce such rows, so HMS gaps do not map one-to-one to zeros. The README does not say how v2.0 beta handles days without HMS data. The provider’s v1.1 code documents its rule for the monitor-level data: dates with missing smoke data count as non-smoke days (an alternative version, filled_smokePM, fills them from 1–2 days before and after).
• Comparing years. The share of county-days classified as smoke days is between 6.5% and 18.5% in every year from 2006 to 2020 (16.0% in 2020), then 28.9% in 2021, 38.4% in 2022, and 47.8% in 2023. Part of this is real smoke, but the smoke-day screen depends on how HMS analysts draw plumes, which changes over time. The jump does not follow the number of plumes: analysts drew about 124 plume polygons per day in 2020 and about 55 in 2023. Also, 2021–2023 exist only in this beta version, so there is no second version to check them against. Compare smoke_pm25 levels or threshold days (days_gt15, days_gt35) rather than raw smoke_days when you compare years.
• Beta version and an older README. The provider calls v2.0 “preliminary and subject to change.” The README in the folder is dated 2022-11-04 and describes the v1.0 county file (2006–2020, .rds and .csv, 2019 county shapefile), not this 2024-09-04 file. Its rule for missing days is the one the script applies.
• Version 1.0 is not shipped. The research project also has v1.0 (2006–2020). Over 2006–2020 the two versions’ monthly means correlate at 0.984, but v1.0 averages 0.434 µg/m³ and v2.0 beta 0.394 µg/m³, and v1.0 lists 2,304,227 smoke county-days against 2,011,678 in v2.0 beta. Never mix versions in one variable.
• Not source-attributed. Smoke from all fires is pooled. The file cannot separate Canadian smoke from U.S. smoke; for that you need fire locations, smoke plumes, and winds from other modules.
• A model estimate. The values are predictions from a model trained on EPA monitors, not measurements. Smoke days come mainly from HMS plumes, which analysts draw from satellite images, and a plume seen from space can sit high above the ground. Outside the West, HMS plumes agree poorly with smoke at the ground (Liu et al. 2024, International Journal of Wildland Fire), so the smoke-day screen is less reliable in the eastern counties that Canadian smoke often reaches.
• Extremes. The highest daily value is 430.80 µg/m³ (Mono County, California, 17 September 2020). The highest monthly mean is 200.18 µg/m³ (Trinity County, California, August 2021).
• Reading the files. read_csv() guesses GEOID as text and smoke_day as logical, but give col_types anyway, as in chunk m05-read-back. All 18 daily files together are 20.4 million rows; read only the years you need.
• Memory and time. The script runs in about 2 minutes, with a peak memory use of about 1.5 GB.

EPA EMBER model runs for 2023

EMBER (Expedited Modeling of Burn Events Results) is a set of air-quality model runs that the U.S. Environmental Protection Agency (EPA) made for the 2023 fire season. EPA ran its Community Multiscale Air Quality model (CMAQ), a computer simulation of how emissions travel, mix, and react in the air, three times over the same 36 km grid: once with all emissions, once with no fire emissions, and once with no Canadian wildfire emissions. Each run gives, for every grid cell and every day from 11 April to 29 September 2023, the 24-hour average PM2.5 (fine particles smaller than 2.5 micrometers across, the main pollutant in wildfire smoke) and the daily maximum 8-hour average ozone (MDA8) near the ground.

Because the three runs differ only in their fires, subtracting one from another gives a model estimate of how much PM2.5 came from all fires, and from Canadian fires alone, in each county on each day. No monitor can split PM2.5 by source this way. But EMBER is a model, not a measurement: its numbers are only as good as its emission estimates and its physics, its fire emissions were built from fire perimeters matched to satellite fire detections (so it is not independent of the satellite fire data used elsewhere in this course), and it covers one season only.

Provider	U.S. Environmental Protection Agency (EPA); data set by Simon et al., archived on Zenodo (record 13737754)
Coverage	11 April – 29 September 2023 (172 days) · contiguous U.S., southern Canada, and most of Mexico on EPA’s “36US3” grid (172 × 148 cells of 36 km × 36 km)
Raw unit	one grid cell = 36 km × 36 km, one value per cell and day, in each of three model runs
Clean unit	one row = one county × one modeled day
Licence and citation	Data: CC BY 4.0. Cite the data, Simon, H., Beidler, J., Baker, K., et al. (2024), Expedited Modeling of Burn Events Results (EMBER) Data Files, Zenodo, https://doi.org/10.5281/zenodo.13737754, and the article, Simon, H., Beidler, J., Baker, K.R., et al. (2025), “Expedited modeling of burn events results (EMBER): A screening-level dataset of 2023 ozone fire impacts in the US,” Data in Brief 58, 111208, https://doi.org/10.1016/j.dib.2024.111208 (the Zenodo description misspells it “Expediated”). The article itself is CC BY-NC.
Documentation	Zenodo record 13737754 (includes the data dictionary shipped here) · Simon et al. 2025, Data in Brief · EPA EMBER page · I/O API file format

Raw files

File (in wildfire_data/raw/)	Size	Format	One row is
ember/2023/HR2DAY_CMAQv54_cb6r5_ae7_aq.36US3.35.basecase2023_2023_O3_PM25_11Apr2023to29Sep2023.nc	35.04 MB	NetCDF-3 (64-bit offset) in I/O API layout	one grid cell × day; base run (all emissions)
ember/2023/HR2DAY_CMAQv54_cb6r5_ae7_aq.36US3.35.basecase2023_nofires_2023_O3_PM25_11Arp2023to29Sep2023.nc	35.04 MB	NetCDF-3 (64-bit offset) in I/O API layout	one grid cell × day; no-fires run (no U.S. or Canadian fire emissions)
ember/2023/HR2DAY_CMAQv54_cb6r5_ae7_aq.36US3.35.basecase2023_noCAfires_2023_O3_PM25_11Apr2023to29Sep2023.nc	35.04 MB	NetCDF-3 (64-bit offset) in I/O API layout	one grid cell × day; no-Canadian-fires run (U.S. fires kept)
ember/2023/EMBER_datadictionary.docx	0.02 MB	Word document	— (the provider’s data dictionary for every file in the Zenodo record)
ember/2023/zenodo_record_13737754.json	0.01 MB	JSON	— (Zenodo’s metadata record for the data set, saved 27 September 2026)

The file names are the provider’s. “11Arp” in the no-fires name is a typo in the original; the file covers the same days as the other two. All three NetCDF files match the MD5 checksums published on Zenodo. (An MD5 checksum is a short fingerprint computed from a file’s bytes; matching fingerprints mean the files are identical.)

Not shipped: the rest of the Zenodo record (fire and other emission inputs, EMBER.EMIS.tar.gz, 25.4 GB; figures, EMBER_figures_table.zip, 0.66 GB; model scripts, EMBER_inputs_and_scripts.zip, 6.4 MB). EPA has also posted EMBER runs for 2021, 2022, 2024, and 2025 (CMAQ 5.5, 1 April – 30 September, about 168 MB per file, on EPA’s public epa-ember Amazon S3 bucket; see the EPA page). Their third run removes Canadian and Mexican fires together, so it cannot isolate Canada. The research project keeps them only as a consistency check, and they are not in this bundle. (In the 2024 and 2025 files the PM2.5 variable is named ATOTIJ, not ATOTIJ_AVG.)

Raw variables

I/O API is the file layout of EPA’s air-quality models: a NetCDF file with fixed dimension names and with the map grid stored in global attributes. The three .nc files share one layout; only the values differ. Their dimensions are TSTEP (172 days, the “unlimited” time dimension), DATE-TIME (2), LAY (1 model layer), VAR (2), ROW (148), and COL (172). That list is the order in which ncdf4 lists the dimensions (what chunk m08-look prints). In the table below, each variable’s dimensions are given in the order of the array that ncvar_get() returns, fastest-changing first (COL for the two data variables). Variable descriptions follow the data dictionary and Simon et al. (2025).

Variable	Dimensions	Units	Description
TFLAG	DATE-TIME × VAR × TSTEP (2 × 2 × 172); integer	<YYYYDDD,HHMMSS>	Time-step flag, one pair per variable and day. Element 1 is the date as YYYYDDD, the year and the day of the year (2023101 = 11 April 2023). Element 2 is the start time as HHMMSS (always 0).
O3_MDA8	COL × ROW × LAY × TSTEP (172 × 148 × 1 × 172); float	ppbV (parts per billion by volume)	Maximum daily 8-hour average ozone in the model’s bottom layer, which the article says was “calculated in accordance with Appendix U to 40 CFR Part 50” (the regulatory MDA8 rule). Variable attribute var_desc = “Max-8-hour”.
ATOTIJ_AVG	COL × ROW × LAY × TSTEP (172 × 148 × 1 × 172); float	ug/m3 (µg/m³)	24-hour average fine particulate matter (PM2.5) in the model’s bottom layer: the sum of all particle species in the model’s two fine-particle size groups, the Aitken mode (“I”, the smallest particles) and the accumulation mode (“J”). Variable attribute var_desc = “Daily-average”.

Global attributes: in I/O API files the map grid and time axis live in global attributes, not in latitude or longitude variables. The values below are from the base file; the other two files are identical except for CTIME and WTIME (the clock time each file was written). Definitions follow the I/O API documentation.

Attribute	Value	Description
IOAPI_VERSION	ioapi-3.2: $Id: init3.F90 247 2023-03-22 …	Version of the I/O API library that wrote the file
EXEC_ID	????????????????	Execution ID of the writing program (not set)
FTYPE	1	File type; 1 = gridded data (GRDDED3)
CDATE, CTIME	2024191, 211903	File creation date (YYYYDDD: 9 July 2024) and time (HHMMSS)
WDATE, WTIME	2024191, 211903	Date and time of the last write
SDATE, STIME	2023101, 0	Date (11 April 2023) and time (00:00:00) of the first time step
TSTEP	240000	Length of a time step as HHMMSS: 24 hours
NTHIK	1	Boundary thickness in cells; used only by boundary files, not by gridded files like these
NCOLS, NROWS, NLAYS	172, 148, 1	Number of grid columns, rows, and layers
NVARS	2	Number of data variables (O3_MDA8, ATOTIJ_AVG; TFLAG is not counted)
GDTYP	2	Map projection type; 2 = Lambert conformal conic (a map projection that unrolls a cone touching the globe at two latitudes; it keeps shapes nearly true across mid-latitude regions such as North America)
P_ALP, P_BET	33, 45	The two latitudes (degrees north) that define the projection cone (standard parallels)
P_GAM	−97	Central meridian (degrees east)
XCENT, YCENT	−97, 40	Longitude and latitude of the point where projected x = 0 and y = 0
XORIG, YORIG	−2,952,000, −2,772,000	x and y (meters) of the lower-left corner of the cell in column 1, row 1
XCELL, YCELL	36,000, 36,000	Cell width and height (meters)
VGTYP	−9999	Vertical coordinate type; −9999 means “missing”
VGTOP	0.999	Model top for sigma coordinates (sigma is the model’s height coordinate that follows the terrain: 1 at the ground, falling toward 0 with height); with VGTYP missing, this value is not documented by the provider
VGLVLS	1, 0.9975	Vertical coordinate values of the layer edges: the single layer here is the model’s bottom layer (sigma 1.0 to 0.9975, a thin slice of air at the ground)
GDNAM	36US3	Grid name: EPA’s 36 km continental U.S. grid
UPNAM	M3XTRACT	Name of the last program that wrote the file (an I/O API tool that extracts subsets)
VAR-LIST	O3_MDA8 ATOTIJ_AVG	Names of the data variables, 16 characters each
FILEDESC	“Concentration file output / Averaged over the synchronization time steps / Timestamp represents beginning computed date/time / Layer mapping (CGRID to AGRID): Layer 1 to 1”	Free-text file description written by CMAQ (padded with blanks)
HISTORY	(empty)	Update history (empty)

zenodo_record_13737754.json is Zenodo’s metadata record for the data set. It is documentation, not data. Its top-level fields:

Field	Type	Description
id, recid	number, string	Zenodo record number (13737754)
doi, doi_url	string	DOI of this version: 10.5281/zenodo.13737754
conceptrecid, conceptdoi	string	Record number (13737753) and DOI that cover all versions of the data set
created, modified, updated	string (ISO date-time, UTC)	When the record was created (2024-09-10) and last changed (2025-01-23)
title	string	Data set title
metadata	object	Descriptive metadata: title, publication_date (2024-09-09), description (HTML), access_right, creators and contributors (names, affiliations, ORCID iDs), keywords, references (the Data in Brief article), language, resource_type, journal, license (cc-by-4.0), relations (versions)
files	array	One entry per file in the record: key (file name), size (bytes), checksum (MD5), id, links (download URL)
links	object	Web and API addresses of the record
revision	number	Revision number of the record (16)
stats	object	View and download counts on the day the record was saved
status, state, submitted	string, string, boolean	Publication status: “published”, “done”, true
owners	array	Zenodo user ID of the record’s owner
swh	object	Software Heritage archive identifiers (empty)

The .docx data dictionary describes each file in the Zenodo record in prose; it has no variables beyond the two listed above.

Cleaning steps

The code makes these decisions:

1. Read the arrays as they are. ncdf4 returns each variable as an array in the order [column, row, day]. Column 1 is the westernmost column and row 1 the southernmost row, so the array never needs to be flipped as long as we index cells by (column, row).
2. Rebuild the model’s map from the file. The PROJ string uses the projection numbers stored in the global attributes. It also treats the Earth as a sphere with a radius of 6,370 km, which is CMAQ’s convention and is not stored in the file. The Census coordinates are on the NAD83 datum; treating them as WGS84 (EPSG:4326) moves them by about a meter, far less than a 36 km cell.
3. Weight by population, not by area. A county’s value is the average of its cells, weighted by where its people live in 2020. Each block group’s center of population (from module m00) is placed in the grid cell that contains it, so a cell over a lake or an empty desert gets no weight. A 36 km cell is much larger than a block group, so assigning each block group to one whole cell loses little.
4. Average first, then subtract. County averages are taken for each run, and fire PM2.5 is the difference of the county averages. Because a weighted mean is linear, this is the same as averaging the cell differences.
5. Define the Canadian share only when there is fire smoke. The share is a ratio of county averages, capped to [0, 1], and set to missing when modeled fire PM2.5, before rounding, is below 0.01 µg/m³.
6. Round only at the end, and keep every county and every modeled day (a complete panel).
7. Keep extreme values, but show them. The file keeps every modeled value, including 135 county-days with more than 100 µg/m³ of fire PM2.5 that came mostly from nearby U.S. fires, which monitors often do not confirm. The last two chunks show where these values are and how to summarize the file with population weights.

Name the three runs. The named vector lets later code say ember_paths[["no_fires"]] instead of repeating a long file name.

# EMBER ran one air-quality model three times for 11 April - 29 September
# 2023. The runs differ only in which fire emissions they include, so the
# difference between two runs is the pollution that the removed fires caused.
#   base            : all emissions, fires included
#   no_fires        : no fire emissions at all (no U.S. and no Canadian fires)
#   no_canada_fires : no Canadian wildfire emissions (U.S. fires still in)
# The file names are the provider's own. The "11Arp" in the no-fires file
# name is a typo in the original name; we keep it so the file matches Zenodo.
ember_files <- c(
  base = "HR2DAY_CMAQv54_cb6r5_ae7_aq.36US3.35.basecase2023_2023_O3_PM25_11Apr2023to29Sep2023.nc",
  no_fires = "HR2DAY_CMAQv54_cb6r5_ae7_aq.36US3.35.basecase2023_nofires_2023_O3_PM25_11Arp2023to29Sep2023.nc",
  no_canada_fires = "HR2DAY_CMAQv54_cb6r5_ae7_aq.36US3.35.basecase2023_noCAfires_2023_O3_PM25_11Apr2023to29Sep2023.nc"
)
ember_paths <- set_names(file.path(raw_dir, "ember", "2023", ember_files), names(ember_files))
stopifnot(all(file.exists(ember_paths)))

Open the base run and print its variables, dimensions, and grid attributes. These are the numbers that tell us where each cell is.

# Open the base run and look inside. The three files have the same layout:
# three variables (TFLAG, O3_MDA8, ATOTIJ_AVG) on 172 columns x 148 rows of
# grid cells, one model layer, and 172 daily time steps.
nc_base <- nc_open(ember_paths[["base"]])
names(nc_base$var)
map_int(nc_base$dim, "len")

# IOAPI files store the map grid in global attributes instead of latitude and
# longitude variables. GDTYP = 2 means a Lambert conformal conic projection.
grid <- ncatt_get(nc_base, 0)
grid[c("GDTYP", "P_ALP", "P_BET", "P_GAM", "XCENT", "YCENT",
       "XORIG", "YORIG", "XCELL", "YCELL", "NCOLS", "NROWS")] |>
  as_tibble()

Turn TFLAG into real dates. make_date(year) is 1 January, so adding the day of the year minus one gives the date.

# TFLAG stores the start of each time step as two integers: the date as
# YYYYDDD (year and day of the year) and the time as HHMMSS. Its dimensions
# are [DATE-TIME, VAR, TSTEP], with one copy per variable; the first is enough.
tflag <- ncvar_get(nc_base, "TFLAG")
nc_close(nc_base)
yyyyddd <- tflag[1, 1, ]
ember_dates <- make_date(yyyyddd %/% 1000) + (yyyyddd %% 1000) - 1

# One time step per day, all at 00:00:00, 11 April to 29 September 2023.
stopifnot(
  length(ember_dates) == 172, all(tflag[2, 1, ] == 0), all(diff(ember_dates) == 1),
  min(ember_dates) == as.Date("2023-04-11"), max(ember_dates) == as.Date("2023-09-29")
)

A small function reads one variable from one file and checks that the file uses the same grid and days as the base run.

# Read one daily variable from one run. ncvar_get() returns an array with the
# dimensions [COL, ROW, TSTEP]; it drops the single model layer (LAY) for us.
# Before reading, check that the run uses the same grid and start date as the
# base run, so cell (col, row) on day t means the same place and day in all.
read_ember <- function(path, var) {
  nc <- nc_open(path)
  on.exit(nc_close(nc))
  same_grid <- map_lgl(
    c("XORIG", "YORIG", "XCELL", "YCELL", "NCOLS", "NROWS", "SDATE", "TSTEP"),
    \(a) identical(ncatt_get(nc, 0, a)$value, grid[[a]])
  )
  x <- ncvar_get(nc, var)
  stopifnot(all(same_grid), !anyNA(x),
            identical(dim(x), c(grid$NCOLS, grid$NROWS, length(ember_dates))))
  x
}

Build the cell-by-day table: 4,378,432 rows, one for each of the 25,456 cells on each of the 172 days, with the three PM2.5 runs and the base ozone side by side. The order of expand_grid() must match the order of the array.

# Turn the arrays into one long table with one row per grid cell and day.
# as.vector() reads an array with its first dimension changing fastest: COL,
# then ROW, then TSTEP. expand_grid() changes its LAST column fastest, so
# listing date, row, col in that order lines the rows up with the values.
cell_day <- expand_grid(
  date = ember_dates,
  row = seq_len(grid$NROWS),
  col = seq_len(grid$NCOLS)
) |>
  mutate(
    pm25_base = as.vector(read_ember(ember_paths[["base"]], "ATOTIJ_AVG")),
    pm25_no_fires = as.vector(read_ember(ember_paths[["no_fires"]], "ATOTIJ_AVG")),
    pm25_no_canada_fires = as.vector(read_ember(ember_paths[["no_canada_fires"]], "ATOTIJ_AVG")),
    o3_mda8_base_ppb = as.vector(read_ember(ember_paths[["base"]], "O3_MDA8"))
  )

# 172 columns x 148 rows x 172 days = 4,378,432 cell-days. Concentrations
# cannot be negative, so the smallest value of each column must be >= 0.
cell_min <- cell_day |>
  summarize(across(pm25_base:o3_mda8_base_ppb, min))
stopifnot(nrow(cell_day) == 172 * 148 * 172, all(cell_min >= 0))
# R prints scientific notation: 1.742e+04 means 17,420 ug/m3, in a cell over
# the Quebec fires. Few such extreme cells hold U.S. residents, so county
# averages are much lower (see m08-hotspots).
summary(select(cell_day, pm25_base:o3_mda8_base_ppb))

The printed summary describes grid cells, not counties, so it includes cells far from any U.S. resident. The largest pm25_base (17,420 µg/m³) is a cell over the fires in northern Quebec on 22 June 2023; in cells where U.S. residents live, the maximum is about 2,320 µg/m³ (on the Oregon–California coast, 27 August), and population weighting brings the largest county value down to 855 µg/m³ (chunk m08-hotspots). The ozone minimum (0.35 ppb) is also a cell in Canada, in northern British Columbia on 23 September, so no county uses it.

Write the grid’s map projection as a PROJ string, a one-line text description of a map projection that sf understands. The grid uses a Lambert conformal conic projection: picture a paper cone set over the globe so that it touches it along two latitudes (33° N and 45° N here) and then unrolled flat.

# To find the grid cell that a place falls in, we need the grid's projection.
# IOAPI's Lambert conformal conic grid has two standard parallels (P_ALP and
# P_BET), a central meridian (P_GAM), and an origin where x = 0 and y = 0
# (XCENT, YCENT). Here P_GAM equals XCENT, so the origin sits on the central
# meridian and no false easting is needed. CMAQ treats the Earth as a sphere
# with a radius of 6,370 km; that number is a CMAQ rule, not stored in the file.
stopifnot(grid$GDTYP == 2, grid$P_GAM == grid$XCENT)
ember_crs <- str_glue(
  "+proj=lcc +lat_1={grid$P_ALP} +lat_2={grid$P_BET} +lat_0={grid$YCENT} ",
  "+lon_0={grid$P_GAM} +x_0=0 +y_0=0 +R=6370000 +units=m +no_defs"
)
ember_crs

Project the block-group centers to the grid and compute each one’s column and row. floor((x - XORIG) / XCELL) + 1 counts how many whole 36 km cells lie between the grid’s western edge and the point.

# Population weights start from the 2020 block-group centers of population
# (module m00). Block groups with no residents are dropped: they would get a
# weight of zero anyway. Each center is projected to the grid's x/y meters.
bg_centers <- read_csv(
  file.path(clean_dir, "census_bg_centers_2020.csv"),
  col_types = cols(bg_geoid = "c", GEOID = "c", .default = "d")
) |>
  filter(pop_2020 > 0)

bg_xy <- bg_centers |>
  st_as_sf(coords = c("lon", "lat"), crs = 4326) |>
  st_transform(st_crs(ember_crs)) |>
  st_coordinates()

# Count whole cells from the grid's lower-left corner (XORIG, YORIG). Column 1
# is the westernmost column and row 1 the southernmost row, the same order as
# the NetCDF arrays, so no flipping is needed.
bg_cells <- bg_centers |>
  mutate(
    col = as.integer(floor((bg_xy[, "X"] - grid$XORIG) / grid$XCELL)) + 1L,
    row = as.integer(floor((bg_xy[, "Y"] - grid$YORIG) / grid$YCELL)) + 1L
  )

# Every populated block group must land inside the 172 x 148 grid.
stopifnot(
  all(between(bg_cells$col, 1, grid$NCOLS)),
  all(between(bg_cells$row, 1, grid$NROWS))
)

Add up population by county and cell, and turn it into weights that sum to 1 in each county. A county touches between 1 and 22 cells (median 3).

# A county's weight on a cell is the share of the county's 2020 population
# whose block-group centers fall in that cell. The weights add up to 1 in
# each county. Lakes and empty land get no weight because nobody lives there.
county_cell_weights <- bg_cells |>
  summarize(pop = sum(pop_2020), .by = c(GEOID, col, row)) |>
  mutate(weight = pop / sum(pop), .by = GEOID)

counties <- read_csv(file.path(clean_dir, "counties_2020.csv"),
                     col_types = cols_only(GEOID = "c"))

# Every one of the 3,108 counties has weights, and they add up to 1.
weight_check <- county_cell_weights |>
  summarize(total = sum(weight), n_cells = n(), .by = GEOID)
stopifnot(
  nrow(counties) == 3108,
  setequal(weight_check$GEOID, counties$GEOID),
  all(abs(weight_check$total - 1) < 1e-9),
  !anyDuplicated(select(county_cell_weights, GEOID, col, row))
)
# How many cells does a county touch? Most touch only a few.
summary(weight_check$n_cells)

Join the weights to the cell-day table and take the weighted mean for each county and day. This step is the slowest (a few seconds) and uses the most memory, about 1.5 GB for the whole script.

# Join each county's weights to the daily values of its cells, then take the
# weighted mean, sum(weight * value), for each county and day. The join is
# many-to-many: a cell can serve several counties, and each county-cell pair
# matches all 172 days.
county_day <- county_cell_weights |>
  select(GEOID, col, row, weight) |>
  inner_join(cell_day, by = c("col", "row"), relationship = "many-to-many") |>
  summarize(
    across(pm25_base:o3_mda8_base_ppb, \(x) sum(weight * x)),
    .by = c(GEOID, date)
  )

Compute fire PM2.5, Canadian-fire PM2.5, and the Canadian share of fire PM2.5.

# Fire PM2.5 is the PM2.5 that disappears when fires are switched off:
#   pm25_fire        = base - no fires          (U.S. and Canadian fires)
#   pm25_canada_fire = base - no Canadian fires (Canadian wildfires only)
# Model chemistry is not perfectly additive, so either difference can be a
# little below zero. The Canadian share of fire PM2.5 is left undefined (NA)
# when fire PM2.5 is below 0.01 ug/m3: a ratio of two tiny numbers is noise.
# Otherwise the share is capped to the range 0 to 1. These values are not
# rounded yet, so a share can be NA where the rounded file shows 0.01.
county_day <- county_day |>
  mutate(
    pm25_fire = pm25_base - pm25_no_fires,
    pm25_canada_fire = pm25_base - pm25_no_canada_fires,
    canada_share_of_fire = if_else(
      pm25_fire >= 0.01,
      pmin(pmax(pm25_canada_fire / pm25_fire, 0), 1),
      NA_real_
    )
  )

Round, order the columns, check the grain, and write the file.

# Round for storage: PM2.5 to 0.01 ug/m3, the share to 0.001, and ozone to
# 0.1 ppb. Differences and the share were computed from unrounded values.
# A tiny negative difference rounds to -0; adding 0 turns it into a plain 0,
# so the file never shows "-0".
dir.create(clean_dir, recursive = TRUE, showWarnings = FALSE)
ember_county_day <- county_day |>
  mutate(
    across(starts_with("pm25"), \(x) round(x, 2) + 0),
    canada_share_of_fire = round(canada_share_of_fire, 3),
    o3_mda8_base_ppb = round(o3_mda8_base_ppb, 1)
  ) |>
  select(GEOID, date, pm25_base, pm25_no_fires, pm25_no_canada_fires,
         pm25_fire, pm25_canada_fire, canada_share_of_fire, o3_mda8_base_ppb) |>
  arrange(GEOID, date)

# Grain: one row per county and modeled day, 3,108 x 172 = 534,576 rows,
# with no duplicated keys. The only missing values are undefined shares.
stopifnot(
  nrow(ember_county_day) == 3108 * 172,
  !anyDuplicated(select(ember_county_day, GEOID, date)),
  setequal(ember_county_day$GEOID, counties$GEOID),
  setequal(ember_county_day$date, ember_dates),
  all(complete.cases(select(ember_county_day, -canada_share_of_fire))),
  all(ember_county_day$pm25_base >= 0), all(ember_county_day$o3_mda8_base_ppb > 0),
  all(between(ember_county_day$canada_share_of_fire, 0, 1), na.rm = TRUE)
)
write_csv(ember_county_day, file.path(clean_dir, "ember_county_day_2023.csv"), na = "")

Two quick summaries. June has by far the most Canadian-fire PM2.5 (5.36 µg/m³ on average across counties), and the largest single values are in north-central Indiana on 28 June 2023.

# A first look. By month: the average across counties (not weighted by
# population) of fire and Canadian-fire PM2.5, and how often the share is NA.
ember_county_day |>
  mutate(month = month(date, label = TRUE)) |>
  summarize(
    fire = mean(pm25_fire), canada_fire = mean(pm25_canada_fire),
    share_na = mean(is.na(canada_share_of_fire)),
    .by = month
  )

# The five county-days with the most modeled Canadian-fire PM2.5.
ember_county_day |>
  slice_max(pm25_canada_fire, n = 5) |>
  select(GEOID, date, pm25_base, pm25_canada_fire, canada_share_of_fire)

The highest values in the file come from U.S. fires, not Canadian smoke. This chunk prints the five largest pm25_base values (all with a Canadian share of 0.055 or less) and counts the county-days with more than 100 µg/m³ of fire PM2.5: 363 are mostly Canadian smoke (May to July, 338 of them in June) and 135 are mostly from U.S. fires.

# The very highest values in the file are not Canadian smoke. They are
# county-days in or next to a grid cell that holds a large U.S. wildfire or
# prescribed burn, so the Canadian share is near 0. Monitors in the same
# county often read far less on those days (see "Checks and cautions"), so
# look at these rows before you average pm25_base or pm25_fire.
ember_county_day |>
  slice_max(pm25_base, n = 5) |>
  select(GEOID, date, pm25_base, pm25_fire, canada_share_of_fire)

# County-days with more than 100 ug/m3 of fire PM2.5, split by whether most
# of that fire PM2.5 came from Canadian fires (share of 0.5 or more).
ember_county_day |>
  filter(pm25_fire > 100) |>
  count(mostly_canada = canada_share_of_fire >= 0.5)

Population-weighted averages describe what the average resident breathed. This chunk reproduces the season averages, the averages for each group of states in the east_split column of counties_2020.csv, and the peak days quoted under “Checks and cautions”.

# What did the average resident breathe? Weight each county by its 2020
# population (pop_2020 in counties_2020.csv). The east_split column of the
# same file puts each state in one of five groups: Great Lakes, Upper
# Midwest, Mid-Atlantic, New England, or Other (all remaining states).
county_pop <- read_csv(file.path(clean_dir, "counties_2020.csv"),
                       col_types = cols_only(GEOID = "c", east_split = "c",
                                             pop_2020 = "d"))
ember_pop <- left_join(ember_county_day, county_pop, by = "GEOID")
stopifnot(nrow(ember_pop) == nrow(ember_county_day), !anyNA(ember_pop$pop_2020))

# Season averages, population-weighted: whole country, then each group.
ember_pop |>
  summarize(across(c(pm25_base, pm25_fire, pm25_canada_fire),
                   \(x) weighted.mean(x, pop_2020)))
ember_pop |>
  summarize(across(c(pm25_base, pm25_fire, pm25_canada_fire),
                   \(x) weighted.mean(x, pop_2020)),
            .by = east_split) |>
  arrange(desc(pm25_canada_fire))

# The three days with the most Canadian-fire PM2.5 for the average resident.
ember_pop |>
  summarize(pm25_canada_fire = weighted.mean(pm25_canada_fire, pop_2020),
            .by = date) |>
  slice_max(pm25_canada_fire, n = 3)

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
ember_county_day_2023.csv	534,576	one county × one modeled day (3,108 counties × 172 days, 11 April – 29 September 2023)	27.5 MB

Clean variables

ember_county_day_2023.csv: sorted by GEOID and date. All values are population-weighted county averages of EMBER grid cells. Ranges are over all rows.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (2020 Census counties; joins to counties_2020.csv)
date	date	—	Modeled day, 2023-04-11 to 2023-09-29 (see the time caution below)
pm25_base	dbl	µg/m³	24-hour average PM2.5 in the base run, all emissions (ATOTIJ_AVG); 0.01 to 855 (median 4.4). Values above about 300 come from cells next to large U.S. fires and are often not seen by monitors: see “Spikes next to U.S. fires” below
pm25_no_fires	dbl	µg/m³	Same, in the run with no fire emissions; 0.01 to 47.1
pm25_no_canada_fires	dbl	µg/m³	Same, in the run with no Canadian wildfire emissions; 0.01 to 854
pm25_fire	dbl	µg/m³	Fire PM2.5 from U.S. and Canadian fires: pm25_base − pm25_no_fires, computed before rounding; −0.60 to 851 (median 0.77). The same U.S.-fire spikes drive its top values
pm25_canada_fire	dbl	µg/m³	Canadian-fire PM2.5: pm25_base − pm25_no_canada_fires, computed before rounding; −0.72 to 220
canada_share_of_fire	dbl	share (0 to 1)	pm25_canada_fire / pm25_fire, capped to [0, 1], computed before rounding; NA (undefined) when the unrounded fire PM2.5 is below 0.01 µg/m³ (16,668 rows, 3.1%; 5,110 of them show pm25_fire = 0.01 after rounding)
o3_mda8_base_ppb	dbl	ppb	Maximum daily 8-hour average ozone in the base run (O3_MDA8); 3.5 to 109.4. Values below 15 ppb (10 county-days, mostly in Oregon and Washington on 31 August 2023) look like model artifacts: see the ozone caution below

Checks and cautions

• Agreement with the project build. R/check_m08.R compares every row with the research project’s population-weighted file (ember2023_county_day.parquet). All 534,576 county-days match: the largest difference is 0.005 µg/m³ for each PM2.5 column, 0.0005 for the share, and 0.05 ppb for ozone, which is exactly the rounding. The share is missing on the same 16,668 county-days in both files.
• Raw copies. The three .nc files and the data dictionary are byte-identical to the project’s copies and match the MD5 checksums published on Zenodo.
• EMBER is a model, not a measurement. On the 92,661 county-days with a PM2.5 monitor (616 counties; EPA’s AQS monitor data, columns pm25_mean and o3_mda8_ppb of aqs_county_day_2006_2025.csv.gz from module m03), EMBER’s base PM2.5 averages 7.18 µg/m³ against 9.72 µg/m³ measured; the correlation is 0.70, and the typical (median) model value is two thirds of the monitor value. On the worst Canadian smoke days it is often too low. On 7 June 2023 it gives 81.5 µg/m³ for the Bronx (36005) and 88.4 for Queens (36081), where monitors averaged 177 and 203. On 17 May 2023, in the 9 North Dakota counties with a monitor, it averages 39 µg/m³ against 191 measured, about 5 times too low. It can also come close: on 28 June, Marion County, Indiana (18097, Indianapolis), the highest monitored Indiana county that day, measured 170 µg/m³ against 190 in the model; the model’s Indiana peak of 224 µg/m³ is in north-central counties that have no monitor. For ozone (128,233 monitored county-days), the model averages 50.0 ppb against 46.0 ppb measured (correlation 0.73). The authors call EMBER a “screening-level” data set.
• Spikes next to U.S. fires. The highest values in the file are not Canadian smoke. All 25 county-days with pm25_base above 300 µg/m³ (13 counties) have a Canadian share below 0.1: they lie in or next to grid cells that hold a large U.S. wildfire or prescribed burn. Satellite fire detections agree: for 24 of the 25, the MODIS file from module m01 (firms_modis_detections_2003_2024.csv.gz) has at least 10 U.S. detections within about 100 km of the county’s center on the same or the previous day (Ocean County, New Jersey, on 12 April has 7). The maximum, 855 µg/m³ in Del Norte County, California (06015) on 24 August 2023, has no monitor. Where there is a monitor, it often reads far less: Atlantic County, New Jersey (34001) on 1 June, 489 µg/m³ modeled against 6.6 measured; Ocean County, New Jersey (34029) on 12 April, 304 against 16.2; Franklin County, Idaho (16041) on 18 September, 218 against 4.5; Lane County, Oregon (41039) on 15 August, 484 against 81.8. Sometimes the monitor agrees (Spokane County, Washington, 53063, on 19 August: 242 against 240). Over the 35 monitored county-days with more than 100 µg/m³ of modeled fire PM2.5 that came mostly from U.S. fires (share below 0.5), the model’s median is 149 µg/m³ against a monitor median of 28; 20 of the 35 monitors read below 35. Over the 93 monitored county-days where that much fire PM2.5 was mostly Canadian, model and monitors agree (medians 135 and 132). So when you summarize pm25_base, pm25_no_canada_fires, or pm25_fire across counties or days, use medians, trim the top values, or flag county-days with very high fire PM2.5 and a low Canadian share (chunk m08-hotspots). pm25_canada_fire is much less affected: its maximum is 220 µg/m³, on the Canadian smoke day of 28 June.
• Very low ozone on a few days. Modeled MDA8 ozone is below 15 ppb on only 10 county-days, 6 of them in Oregon and Washington on 31 August 2023: Multnomah County, Oregon (41051) 3.5 ppb and Clark County, Washington (53011) 4.2 ppb, where monitors read 19 and 26 ppb. The days before and after look normal. Treat these values as model artifacts.
• What the runs remove. The no-Canadian-fires run drops the Canadian fire emissions sector, which holds all Canadian fires inside the grid (the provider calls them Canadian wildfires); U.S. wildfires and prescribed burns stay in. The no-fires run drops all U.S. and Canadian wild and prescribed fire emissions. Mexican fires and agricultural burning are in none of the runs.
• Fires outside the grid. The grid reaches about 55° N at its corners and about 62° N at its center (longitude −97°), so, in the authors’ words, it covers “most of the Canadian wildfires” of 2023, but not all. Smoke that blows in across the edge of the grid comes from the same boundary data in all three runs, so it cancels out in the differences and is counted as “no-fire” PM2.5.
• Shared inputs with satellite fire data. EMBER’s fire emissions come from Canada’s burned-area perimeters (National Burned Area Composite, NBAC) and U.S. fire perimeters (WFIGS), matched to NOAA Hazard Mapping System (HMS) satellite fire detections and run through the U.S. Forest Service BlueSky emissions system. So EMBER fire PM2.5 is not independent of the satellite fire and smoke data in other modules; agreement between them is partly built in. The 2023 fire emissions were preliminary, and the non-fire emissions are typical 2018 days, not actual 2023 days.
• Negative values are real model output. Chemistry is not additive, so removing fires can slightly raise PM2.5 in some places. In the file, pm25_fire is below 0 on 2,249 county-days (minimum −0.60 µg/m³) and pm25_canada_fire on 6,331 (minimum −0.72). Before rounding, 4,306 and 29,461 values were below 0; the tiny ones (between −0.005 and 0) round to 0. Do not take logs without handling them.
• The share. Missing (NA) means “undefined: less than 0.01 µg/m³ of fire PM2.5”, not “not observed”. It is most common in April (9.1% of county-days) and May (6.3%). The cap changed 30,071 county-days: 23,011 had a negative Canadian part (share set to 0) and 7,060 had a Canadian part larger than total fire PM2.5 (share set to 1). After rounding to 0.001, more rows show the end values: 27,534 rows show 0 and 12,253 show 1.
• Rounding. The differences and the share were computed before rounding. So pm25_fire can differ by 0.01 from pm25_base − pm25_no_fires recomputed from the rounded columns (133,370 rows; 127,516 rows for pm25_canada_fire), and 5,110 rows show pm25_fire = 0.01 with a missing share because the unrounded value was just below 0.01.
• What a “day” is. Each value belongs to a model day. The article says MDA8 ozone follows the regulatory rule (Appendix U), which uses local standard time; the provider does not say whether the 24-hour PM2.5 averages use local days or UTC days (UTC is Coordinated Universal Time, 5 to 8 hours ahead of U.S. local standard time, or 4 to 7 hours ahead during daylight saving time). The model itself ran in UTC, starting each day at 00:00 UTC (run script in the Zenodo record). The daily files were made with CMAQ’s hr2day tool (the “HR2DAY” in the file names), where one setting (USELOCAL, default “no”) switches every variable in a run to local time. Ozone and PM2.5 sit in one file, so they probably came from one run of the tool. The 2021 EMBER files, written directly by hr2day (their UPNAM attribute is “hr2day”), hold ozone and PM2.5 variables together, and the 2023 files look like extracts of such a file (UPNAM = “M3XTRACT”). If so, the PM2.5 days are local-standard-time days as well. This is our inference, not a provider statement. When you compare with monitor data, which use local days, expect possible offsets of a few hours.
• Coarse grid. A 36 km cell is larger than many counties. Only 18 counties have values identical to another county’s on every day: they take all their population from the same single cell. Sixteen of them are in Virginia, most of them small independent cities such as Alexandria and Manassas; the other two are Union County, New Jersey, and Staten Island (Richmond County, New York). Many more neighboring counties have nearly identical values, and city-scale differences are invisible.
• Big picture. Averaged over the season and weighted by population, modeled PM2.5 is 7.40 µg/m³, fire PM2.5 2.55 µg/m³, and Canadian-fire PM2.5 1.87 µg/m³. The national peak is 7 June 2023 (21.7 µg/m³ of Canadian-fire PM2.5, population-weighted). By the east_split groups of states in counties_2020.csv (also weighted by pop_2020), season Canadian-fire PM2.5 is 4.17 µg/m³ in the Great Lakes group, 3.41 in Upper Midwest, 3.23 in Mid-Atlantic, 1.59 in New England, and 0.94 in “Other” (all remaining states). Chunk m08-pop-weighted reproduces these numbers.
• Running time. The script takes 15 to 40 seconds, depending on the computer and how busy it is, and needs about 1.5 GB of memory (peak), most of it in the m08-county-day step. On a laptop with 4 GB of memory, close other programs before you run it.

EPA AQS air-quality monitors

The U.S. Environmental Protection Agency (EPA) collects the measurements of ground-level air-quality monitors in its Air Quality System (AQS). State, local, and tribal air agencies run the monitors and report to EPA, and EPA publishes “pre-generated” daily summary files on its AirData website. We use two pollutants: fine particulate matter (PM2.5, particles smaller than 2.5 micrometers), measured in micrograms per cubic meter (µg/m³), and ozone, measured in parts per million (ppm). For PM2.5 we use parameter code 88101, the FRM/FEM monitors: federal reference methods and federal equivalent methods, the instruments EPA accepts for checking whether an area meets its air-quality standards.

For this project, AQS is the measured truth on the ground. It shows how bad the air actually was at a monitor on a given day, for example when Canadian smoke covered New York and Pennsylvania in June 2023, and it can be used to check model-based smoke estimates such as the Childs et al. smoke PM2.5 (model estimates of the part of PM2.5 that comes from wildfire smoke; see the section “Childs et al. smoke PM2.5 (Stanford ECHO Lab)”). It cannot tell you where the pollution came from: a monitor measures all PM2.5 and all ozone, from smoke, traffic, and industry alike. It also covers only places with monitors: 1,080 of the 3,108 counties have at least one site, and many monitors do not report every day.

Provider	U.S. Environmental Protection Agency, Air Quality System (AQS), through the AirData pre-generated daily summary files. File version generated by EPA on 2026-06-25; downloaded on 2026-09-27
Coverage	1 January 2006 – 31 December 2025 · raw files: all U.S. states and territories plus a few sites in Mexico and Canada; clean files: the contiguous U.S. and DC (2,773 sites in 1,080 counties)
Raw unit	one row = one monitor (site × pollutant × POC) on one local day, for one sample duration, one pollutant standard, and one event type
Clean unit	one row = a site (site list); a site-day (PM2.5 and ozone files); a county-day with at least one reading; a county-month with at least one reading
Licence and citation	Public domain: EPA states that AQS ambient monitoring data “are public domain” and may be used without a request. EPA’s suggested citation: US Environmental Protection Agency. Air Quality System Data Mart [internet database] available via https://www.epa.gov/outdoor-air-quality-data. Accessed September 27, 2026. Also name the file version (generated 2026-06-25)
Documentation	AirData file formats; AirData pre-generated files; PM2.5 data advisory on T640 monitors (2024); AirData frequent questions

Raw files

File (in wildfire_data/raw/)	Size	Format	One row is
aqs/daily_88101_<year>__asof20260625.zip (20 files, 2006–2025)	1.6–9.9 MB each; 104.2 MB in total	zip archive with one comma-separated text file (daily_88101_<year>.csv, 52–322 MB unzipped), header row, text fields in double quotes	a PM2.5 monitor-day for one sample duration and event type (139,281 rows in 2006 to 863,466 in 2022; 9,203,911 in total)
aqs/daily_44201_<year>__asof20260625.zip (20 files, 2006–2025)	4.4–4.9 MB each; 92.9 MB in total	zip archive with one comma-separated text file (daily_44201_<year>.csv, 123–140 MB unzipped), header row, text fields in double quotes	an ozone monitor-day for one event type (355,145 to 404,185 rows per year; 7,691,814 in total)
aqs/file_list__dl20260927.csv	0.09 MB	comma-separated text, header row	one file on EPA’s AirData download page (2,049 rows), with its row count, size, and date

The names keep EPA’s file names (daily_88101_2023.zip) and add __asof<date>, the date EPA generated the file, and __dl<date>, the date the project downloaded it. readr reads the CSV inside each zip directly, so you never need to unzip. Unzipped, the PM2.5 files alone would take 3.5 GB (sizes in this section are decimal: 1 MB = 1,000,000 bytes, as Finder on a Mac shows them). The number of PM2.5 rows grew sixfold from 2006 to 2022. Continuous monitors joined the network after 2008, and each one writes two rows per day (1 HOUR and 24-HR BLK AVG); a Teledyne T640 with EPA’s corrected data adds a second POC, so up to four rows per day. The clean PM2.5 site-days grew only about 2.3 times (125,266 in 2006, 283,157 in 2022). The number of ozone rows hardly changed:

Year	PM2.5 rows	Ozone rows
2006	139,281	355,145
2007	145,631	362,148
2008	147,108	364,607
2009	191,009	370,320
2010	230,028	378,545
2011	251,767	383,872
2012	275,331	386,438
2013	317,591	389,281
2014	349,157	388,524
2015	373,478	386,361
2016	408,270	392,841
2017	459,536	404,185
2018	556,441	402,908
2019	654,337	392,961
2020	704,092	395,536
2021	790,468	394,384
2022	863,466	387,156
2023	847,162	388,607
2024	740,924	386,972
2025	758,834	381,023

Raw variables

All 40 daily files (PM2.5 and ozone) have the same 29 columns. Descriptions follow EPA’s AirData file-format page; value ranges and codes are those found in the 40 files. In the file, text and dates are in double quotes and numbers are not.

Variable	Type	Description
State Code	text, 2 characters	FIPS code (Federal Information Processing Standards: the official numeric codes for U.S. states and counties) of the state where the monitor is, with its leading zero ("01" = Alabama). Besides the 50 states and DC, the files contain 02 Alaska, 15 Hawaii, 72 Puerto Rico, 78 Virgin Islands, 80 (“Country Of Mexico”: sites in Baja California, Chihuahua, and Sonora), and CC (“Canada”, sites in Saskatchewan, 2006 only)
County Code	text, 3 characters	FIPS code of the county where the monitor is. State Code + County Code is the 5-character county GEOID. Connecticut sites carry the 8 county codes of the 2020 Census (09001–09015)
Site Num	text, 4 characters	Number of the site within its county. State Code + County Code + Site Num is the 9-character site ID (360810124 = Queens College, New York)
Parameter Code	text, 5 digits	AQS code of what is measured: 88101 = PM2.5 by FRM/FEM monitors; 44201 = ozone. One value per file
POC	integer	Parameter Occurrence Code: tells apart several instruments or data streams that measure the same pollutant at the same site. EPA notes that the numbering is “somewhat arbitrary”. 1–33 in the PM2.5 files, 1–9 in the ozone files
Latitude	decimal number	Site latitude in decimal degrees north of the equator
Longitude	decimal number	Site longitude in decimal degrees east of the prime meridian (negative = west)
Datum	text	Geodetic datum of the coordinates: NAD83 or WGS84 (they differ by about a meter)
Parameter Name	text	Name of the parameter: PM2.5 - Local Conditions or Ozone. “Local conditions” means the air volume is taken at the site’s actual temperature and pressure
Sample Duration	text	How long air passes through the instrument before a value is measured. PM2.5: 24 HOUR (a filter collects particles for 24 hours and is weighed: one value per day), 1 HOUR (a continuous monitor; the row summarizes the day’s hourly values), 24-HR BLK AVG (a 24-hour block average that EPA calculates from the hourly values). Ozone: 8-HR RUN AVG BEGIN HOUR (8-hour running averages, each labeled by the hour it begins)
Pollutant Standard	text	The air-quality standard whose rules EPA used to build the day’s summary. PM2.5: PM25 24-hour 2012 (blank on 1 HOUR rows). Ozone: Ozone 8-hour 2015 on every row
Date Local	date text (YYYY-MM-DD)	Calendar day of the summary: midnight to midnight in local standard time at the monitor (no daylight saving time)
Units of Measure	text	Micrograms/cubic meter (LC) for PM2.5 (µg/m³ at local conditions); Parts per million for ozone
Event Type	text	Whether data measured during exceptional events are in the day’s summary. EPA defines exceptional events as unusual or natural events, such as wildfires or high winds, that air agencies cannot reasonably control. None = no data were flagged; Included = flagged data are included; Excluded = flagged data are removed. A flagged day can have both an Included and an Excluded row. EPA’s page also lists “Concurred Events Excluded”, which does not occur in these files
Observation Count	integer	Number of observations behind the day’s values. PM2.5 1 HOUR rows: 1–24 hourly values; 24 HOUR rows: 1 (2 in a few rows); 24-HR BLK AVG rows: 1. Ozone: number of 8-hour averages, 17 on 96% of rows (1–34)
Observation Percent	decimal number	Observations as a percentage of those scheduled for the day (4–200; above 100 in 101 rows). 24-HR BLK AVG rows always show 100
Arithmetic Mean	decimal number	Average value for the day, in Units of Measure. For ozone rows it is the average of the day’s 8-hour averages. PM2.5: −9.75 to 834 µg/m³
1st Max Value	decimal number	Highest value of the day. For ozone rows this is the maximum daily 8-hour average (MDA8), in ppm. For PM2.5 24-hour rows it equals Arithmetic Mean
1st Max Hour	integer	Hour (0–23, local standard time) when the highest value occurred. For ozone rows, the hour the highest 8-hour period begins (7–23 in these files). 0 on PM2.5 24-hour rows, except 2,843 24 HOUR rows
AQI	integer	Air Quality Index for the day for this pollutant (0 = good; higher is worse). Blank on PM2.5 1 HOUR rows and in one ozone row
Method Code	text, 3 digits	Code of the sampling and analysis method. 32 codes in the PM2.5 files and 21 in the ozone files; blank in 241 ozone rows. Examples: 145 a filter sampler (R & P 2025); 170 and 209 Met One BAM monitors; 236/238 Teledyne T640/T640X as first reported; 736/738 the same T640/T640X data as corrected by EPA in 2024; 636/638 T640/T640X with network data alignment
Method Name	text	Short description of the method’s equipment and procedure
Local Site Name	text	Name the operating agency gives the site; blank on 164,582 PM2.5 rows and 177,909 ozone rows
Address	text	Approximate street address of the site
State Name	text	Name of the state (or Country Of Mexico, Canada)
County Name	text	Name of the county
City Name	text	City whose legal limits contain the site, or Not in a city
CBSA Name	text	Core-based statistical area (metropolitan or micropolitan area) containing the site; blank outside CBSAs
Date of Last Change	date text (YYYY-MM-DD)	Date any number in the row was last updated in AQS (2023-02-05 to 2026-06-25 in these files)

The file list file_list__dl20260927.csv has four columns. EPA’s download page describes its contents.

Variable	Type	Description
Filename	text	Name of a file on the AirData download page, as EPA names it (daily_88101_2023.zip, without our __asof tag)
Rows	integer	Number of data rows in that file. The script checks every read against it
Size	integer	Size of the zipped file in kilobytes (KB), as shown on the download page
Created	date (YYYY-MM-DD)	Date EPA generated the file (“As of” on the download page); 2026-06-25 for all 2,049 files

Cleaning steps

The script R/m03_aqs.R makes these decisions:

1. Read the zips directly, with types. readr reads the CSV inside each zip. Codes stay text so leading zeros survive, and every read is checked against EPA’s row count.
2. One year at a time. The raw PM2.5 files hold 9.2 million rows (3.5 GB unzipped). Each year is read, reduced to site-days, and dropped before the next.
3. (a) Event Type: keep None + Included. This is the “events included” series: what the monitors measured, smoke days included. Excluded rows are a second copy of some flagged days with the flagged data removed, so keeping them would count those days twice. EPA writes an Excluded row only for some flags: in 2023 there are 207,372 Included PM2.5 rows but only 279 Excluded rows.
4. (b) One value per instrument and day. The same PM2.5 instrument-day can appear once per sample duration, and 24 HOUR and 24-HR BLK AVG rows also carry a pollutant standard while 1 HOUR rows do not. Older (and maybe future) AirData files repeat each 24-hour row once per PM2.5 standard. This vintage has a single standard (PM25 24-hour 2012), so no row is repeated; a stopifnot() check stops the script if a second standard appears, and then you would filter to one standard first. The script keeps one row per instrument-day: a 24 HOUR filter sample first, then the 24-HR BLK AVG, then the 1 HOUR mean. A day built from hourly values needs at least 75% of the hours (18 of 24), as in the project build. EPA writes 24-HR BLK AVG rows only for days that meet its own completeness rule, and their Observation Percent is always 100, so in practice our 75% filter acts only on the 1 HOUR fallback.
5. Average the POCs. The site-day value is the mean over the site’s POCs; n_poc says how many were averaged.
6. Ozone: the 2015 standard’s MDA8. Keep Ozone 8-hour 2015 rows, take 1st Max Value (the maximum daily 8-hour average), and multiply by 1,000 to go from ppm to parts per billion (ppb).
7. County from the AQS code. A site’s county (GEOID) is its AQS state + county code when that code is one of the 3,108 counties in counties_2020.csv. Otherwise the script uses the county that contains the site’s coordinates (point-in-polygon). Sites outside the contiguous U.S. are dropped (76 sites).
8. Keep negative PM2.5 values and flag them. AQS accepts small negative daily means as instrument noise near zero. Dropping them or setting them to 0 would push averages up.
9. Round, then aggregate. Site-day values are rounded to 2 decimals. The county-day value is the simple mean over the county’s sites that reported that day, and the county-month file summarizes the county-day file.
10. Sparse files. A county-day or county-month exists only if some monitor reported. No row means no reading, not zero.
11. Write big files one year at a time, and free memory early. The three .csv.gz files are written year by year into the same file, so the rows are sorted by year first. Objects that later steps do not need are removed with rm() as soon as possible. This keeps the script’s memory use near 2 GB.

The first chunk lists the zips and checks that each pollutant has exactly one file for every year from 2006 to 2025.

# AirData has one zip per pollutant and year. Parameter code 88101 is PM2.5
# measured by FRM/FEM monitors (federal reference or equivalent methods: the
# instruments EPA accepts for checking its air-quality standards); 44201 is
# ozone. "asof" in the name is the date EPA generated the file.
aqs_dir <- file.path(raw_dir, "aqs")
pm25_files <- list.files(aqs_dir, pattern = "^daily_88101_\\d{4}__asof\\d{8}\\.zip$",
                         full.names = TRUE)
o3_files <- list.files(aqs_dir, pattern = "^daily_44201_\\d{4}__asof\\d{8}\\.zip$",
                       full.names = TRUE)

# The year is the 4 digits after the parameter code. list.files() sorts the
# names, so the files come in year order.
file_year <- \(f) as.integer(str_match(basename(f), "^daily_\\d{5}_(\\d{4})")[, 2])
stopifnot(
  identical(file_year(pm25_files), 2006:2025),
  identical(file_year(o3_files), 2006:2025)
)

EPA’s file list gives the number of data rows in each AirData file. We keep the rows for the daily PM2.5 and ozone files.

# EPA's index of every AirData file, saved on the day the project downloaded
# the zips. Its Rows column is the number of data rows in each file, so we
# can check that every zip is read completely.
file_list <- read_csv(
  file.path(aqs_dir, "file_list__dl20260927.csv"),
  col_types = cols(Filename = "c", Rows = "i", Size = "d", Created = "D")
)
expected_rows <- file_list |>
  filter(str_detect(Filename, "^daily_(88101|44201)_")) |>
  select(Filename, Rows)

One function reads any of the 40 zips. It reads 15 of the 29 columns with explicit types, checks the row count against EPA’s list, and gives the columns short snake_case names.

# The daily files have 29 columns; we read the 15 we need. Codes stay text so
# that leading zeros survive ("01" is Alabama). Date Local is the calendar
# day at the monitor in local standard time (no daylight saving time).
aqs_cols <- cols_only(
  `State Code` = "c", `County Code` = "c", `Site Num` = "c", POC = "i",
  Latitude = "d", Longitude = "d", `Local Site Name` = "c",
  `Sample Duration` = "c", `Pollutant Standard` = "c", `Date Local` = "D",
  `Event Type` = "c", `Observation Percent` = "d",
  `Arithmetic Mean` = "d", `1st Max Value` = "d", `Method Code` = "c"
)

read_aqs <- function(file) {
  # readr reads the one CSV inside a .zip directly; no need to unzip.
  aqs <- read_csv(file, col_types = aqs_cols, progress = FALSE)
  # A broken download can lose rows without an error; compare with EPA's count.
  zip_name <- str_remove(basename(file), "__asof\\d{8}")
  stopifnot(identical(nrow(aqs), expected_rows$Rows[expected_rows$Filename == zip_name]))
  aqs |>
    transmute(
      site_id = str_c(`State Code`, `County Code`, `Site Num`),  # 2 + 3 + 4 characters
      aqs_geoid = str_c(`State Code`, `County Code`),            # county code used by AQS
      poc = POC, latitude = Latitude, longitude = Longitude,
      local_site_name = `Local Site Name`,
      duration = `Sample Duration`, standard = `Pollutant Standard`,
      date = `Date Local`, event_type = `Event Type`, obs_pct = `Observation Percent`,
      daily_mean = `Arithmetic Mean`, daily_max = `1st Max Value`,
      method = `Method Code`
    )
}

Before cleaning all twenty years, look at 2023. The counts show the three Event Type values and the three PM2.5 sample durations. The last table shows every row for one site on one smoky day: five rows for three POCs.

# Look at one year before cleaning all twenty. In 2023, smoke from Canadian
# fires covered much of the eastern U.S.
pm25_2023 <- read_aqs(pm25_files[file_year(pm25_files) == 2023])
count(pm25_2023, event_type)
count(pm25_2023, duration, standard)

# All rows for one site on one smoky day: Queens College, New York City,
# June 6, 2023. POC (parameter occurrence code) numbers the PM2.5 data
# streams at the site. POC 1 is a filter sampler (method 145). POCs 4 and 24
# are one Teledyne T640 instrument reported twice: as first measured (method
# 236) and as corrected by EPA in 2024 (method 736).
pm25_2023 |>
  filter(site_id == "360810124", date == as.Date("2023-06-06")) |>
  select(poc, method, duration, standard, event_type, obs_pct, daily_mean)

This function turns one year of raw PM2.5 rows into one value per site and day. It applies teaching points (a) and (b) above, and stopifnot() confirms that no instrument has two rows for the same day and duration once Excluded rows are gone.

# Preference order when one instrument reports the same day more than once:
# a 24-hour filter sample, then EPA's 24-hour average of the hourly readings
# ("24-HR BLK AVG"), then the plain mean of the hourly readings ("1 HOUR").
duration_rank <- c("24 HOUR" = 1, "24-HR BLK AVG" = 2, "1 HOUR" = 3)

pm25_site_day <- function(aqs) {
  pm25_rows <- aqs |>
    # (a) Event Type: "None" = no exceptional event was flagged; "Included" =
    # flagged data (for example, wildfire smoke) kept in the day's value;
    # "Excluded" = the same day with the flagged data removed. None + Included
    # keeps smoky days as they were measured.
    filter(event_type %in% c("None", "Included")) |>
    # A day built from hourly readings needs at least 75% of them (18 of 24).
    # EPA writes 24-HR BLK AVG rows only for days that meet its own
    # completeness rule, so they always show 100 here: in practice this filter
    # acts only on the 1 HOUR fallback. 24-hour samples are one reading, so
    # they are kept as they are.
    filter(duration == "24 HOUR" |
             (duration %in% names(duration_rank) & obs_pct >= 75))
  # Within None + Included, an instrument has at most one row per day and
  # sample duration, so the next step never has to choose between events.
  # Older (and maybe future) AirData files repeat each 24-hour row once per
  # PM2.5 standard. This vintage has a single standard ("PM25 24-hour 2012"),
  # so no row is repeated. If a second standard appears, this check stops the
  # script: then filter to one value of `standard` first.
  stopifnot(nrow(distinct(pm25_rows, site_id, poc, date, duration)) == nrow(pm25_rows))
  pm25_rows |>
    # (b) Sort so the preferred duration comes first, then keep the first row
    # of each instrument-day.
    mutate(rank = duration_rank[duration]) |>
    arrange(site_id, poc, date, rank) |>
    distinct(site_id, poc, date, .keep_all = TRUE) |>
    # Average the site's POCs: one row per site and day. Like the project
    # build, we keep every method code, so a T640 with original and corrected
    # data counts twice (see "Checks and cautions" in the guide).
    summarize(pm25 = mean(daily_mean), n_poc = n(), .by = c(site_id, date))
}

Apply the rules to 2023 and look at the Queens College day again. The count of n_poc shows that most site-days have one or two POCs.

# Try the rules on 2023. The Queens College day becomes one row: the mean of
# its three POCs, (58.6 + 101 + 99.5) / 3 = 86.4 µg/m³.
pm25_days_2023 <- pm25_site_day(pm25_2023)
filter(pm25_days_2023, site_id == "360810124", date == as.Date("2023-06-06"))
count(pm25_days_2023, n_poc)
rm(pm25_2023)  # free memory before reading all years

Now run every year. map() returns, for each year, the year’s site list, its Event Type counts, and its site-days; list_rbind() stacks each piece across years.

# Run the same steps on every year. Each raw file is read, used, and dropped,
# so only one year of raw rows is in memory at a time (2023 alone has 847,162).
pm25_by_year <- map(pm25_files, \(f) {
  aqs <- read_aqs(f)
  list(
    sites = distinct(aqs, site_id, aqs_geoid, latitude, longitude, local_site_name),
    events = count(aqs, year = year(date), event_type),
    days = pm25_site_day(aqs)
  )
})
pm25_days_all <- map(pm25_by_year, "days") |> list_rbind()
pm25_sites <- map(pm25_by_year, "sites") |> list_rbind()
pm25_events <- map(pm25_by_year, "events") |> list_rbind()
rm(pm25_by_year)

# Raw rows by Event Type and year. "Included" rows rise sharply in 2018-2023.
# Most of that rise is the original T640 data (method codes 236 and 238),
# which carries an event flag on almost every day, not extra smoke flags.
pm25_events |>
  pivot_wider(names_from = event_type, values_from = n, values_fill = 0) |>
  print(n = 20)

Ozone needs no duration ranking, because each instrument has a single 8-hour-standard row per day. The function keeps those rows and converts ppm to ppb.

# Ozone rows follow the 2015 8-hour standard. For those rows, "1st Max Value"
# is the day's highest 8-hour running average (MDA8: maximum daily 8-hour
# average), in parts per million (ppm). 1 ppm = 1,000 parts per billion (ppb).
# Every ozone row in these files uses this standard; filtering on it makes
# the choice explicit and protects us if EPA adds rows for other standards.
o3_site_day <- function(aqs) {
  o3_rows <- aqs |>
    filter(event_type %in% c("None", "Included"),
           standard == "Ozone 8-hour 2015")
  # One row per instrument and day, so no reading is counted twice.
  stopifnot(nrow(distinct(o3_rows, site_id, poc, date)) == nrow(o3_rows))
  o3_rows |>
    summarize(o3_mda8_ppb = 1000 * mean(daily_max), n_poc = n(),
              .by = c(site_id, date))
}

The same loop for the ozone files.

# Same pattern as for PM2.5: read one year, keep its site list, Event Type
# counts, and site-days, then drop the raw rows.
o3_by_year <- map(o3_files, \(f) {
  aqs <- read_aqs(f)
  list(
    sites = distinct(aqs, site_id, aqs_geoid, latitude, longitude, local_site_name),
    events = count(aqs, year = year(date), event_type),
    days = o3_site_day(aqs)
  )
})
o3_days_all <- map(o3_by_year, "days") |> list_rbind()
o3_sites <- map(o3_by_year, "sites") |> list_rbind()
o3_events <- map(o3_by_year, "events") |> list_rbind()
rm(o3_by_year)

o3_events |>
  pivot_wider(names_from = event_type, values_from = n, values_fill = 0) |>
  print(n = 20)

Give each site a county. The printed tables show how many sites kept their AQS county code, which sites lie outside the contiguous U.S., and which sites have coordinates in a different county than their code.

# One row per site. A site has the same coordinates and name in every year
# and in both pollutants' files, so distinct() leaves one row per site_id.
site_info <- bind_rows(pm25_sites, o3_sites) |> distinct()
stopifnot(!anyDuplicated(site_info$site_id))

county_ids <- read_csv(file.path(clean_dir, "counties_2020.csv"),
                       col_types = cols(GEOID = "c"), col_select = GEOID)$GEOID
county_shapes <- read_sf(file.path(clean_dir, "counties_2020.gpkg")) |>
  select(geoid_pip = GEOID)

# Point-in-polygon: which 2020 county contains each site? Latitude and
# longitude are in NAD83 or WGS84 (see Datum), which differ by about a meter,
# so we treat both as WGS84 (EPSG:4326).
sites_geo <- site_info |>
  st_as_sf(coords = c("longitude", "latitude"), crs = 4326, remove = FALSE) |>
  st_join(county_shapes, join = st_within) |>
  st_drop_geometry() |>
  # Use the AQS county code when it is a 2020 county; otherwise use the county
  # that contains the site. These files still give Connecticut sites the 8
  # 2020 county codes, so the fallback would matter only if EPA switched them
  # to the 2022 planning-region codes (09110-09190). Sites outside the
  # contiguous U.S. fall in no county and get NA.
  mutate(GEOID = if_else(aqs_geoid %in% county_ids, aqs_geoid, geoid_pip))
stopifnot(nrow(sites_geo) == nrow(site_info))  # no site landed in two counties

# Sites by rule: code kept, or no county (outside the contiguous U.S.)
count(sites_geo, code_is_county = aqs_geoid %in% county_ids, has_GEOID = !is.na(GEOID))
# Sites with no GEOID, by AQS state code (02 AK, 15 HI, 72 PR, 78 VI,
# 80 Mexico, CC Canada)
sites_geo |> filter(is.na(GEOID)) |> count(state_code = str_sub(site_id, 1, 2))
# Sites whose coordinates fall in a different county than their code says
sites_geo |>
  filter(!is.na(GEOID), !is.na(geoid_pip), geoid_pip != GEOID) |>
  select(site_id, local_site_name, GEOID, geoid_pip)

# The county shapes and the per-pollutant site lists are no longer needed.
# rm() deletes them and gc() hands their memory back right away.
rm(county_shapes, site_info, pm25_sites, o3_sites)
invisible(gc())

Define a helper that writes a big table one year at a time, then attach the county, round, flag negative values, check the grain, and write the PM2.5 site-day file.

# A helper that writes a big table one year at a time. Sorting millions of
# rows and writing them as one block both need a full extra copy in memory
# (write_csv() builds the text before compressing it): about 1 GB more for
# these files. One year at a time needs only a small fraction of that. Each
# year's rows are sorted by `key` (site_id or GEOID) and date, then added to
# the end of the same .csv.gz file (append = TRUE; the header is written only
# for the first year). read_csv() reads the result as one table, sorted by
# year, then key, then date.
write_csv_by_year <- function(data, key, path) {
  years <- year(data$date)
  for (y in sort(unique(years))) {
    data[years == y, ] |>
      arrange(.data[[key]], date) |>
      write_csv(path, na = "", append = y != min(years))
  }
}

# Keep sites in the contiguous U.S. and attach their county. Averages are
# rounded to 2 decimals; every later file is built from these rounded values.
site_geoid <- sites_geo |> filter(!is.na(GEOID)) |> select(site_id, GEOID)

pm25_site_day_out <- pm25_days_all |>
  inner_join(site_geoid, by = "site_id") |>
  mutate(pm25 = round(pm25, 2)) |>
  # Negative daily means are instrument noise near zero. AQS accepts them,
  # and dropping them or setting them to 0 would bias averages upward, so we
  # keep them and flag them.
  transmute(site_id, GEOID, date, pm25, n_poc, pm25_negative = pm25 < 0)
rm(pm25_days_all)  # the unrounded copy is no longer needed; free the memory
invisible(gc())

# Grain: one row per site and day; no missing values. n_distinct() counts
# the distinct site-day pairs without building a second table.
stopifnot(
  n_distinct(pm25_site_day_out$site_id, pm25_site_day_out$date) ==
    nrow(pm25_site_day_out),
  !anyNA(pm25_site_day_out),
  all(year(pm25_site_day_out$date) %in% 2006:2025),
  all(between(pm25_site_day_out$pm25, -10, 1000)),
  all(pm25_site_day_out$n_poc >= 1)
)
write_csv_by_year(pm25_site_day_out, "site_id",
                  file.path(clean_dir, "aqs_pm25_site_day_2006_2025.csv.gz"))

The same steps for ozone.

o3_site_day_out <- o3_days_all |>
  inner_join(site_geoid, by = "site_id") |>
  transmute(site_id, GEOID, date, o3_mda8_ppb = round(o3_mda8_ppb, 2), n_poc)
rm(o3_days_all)
invisible(gc())

# Grain: one row per site and day; no missing values; ozone in a plausible
# range (a reading slightly below zero is instrument noise, as for PM2.5).
stopifnot(
  n_distinct(o3_site_day_out$site_id, o3_site_day_out$date) ==
    nrow(o3_site_day_out),
  !anyNA(o3_site_day_out),
  all(year(o3_site_day_out$date) %in% 2006:2025),
  all(between(o3_site_day_out$o3_mda8_ppb, -5, 250)),
  all(o3_site_day_out$n_poc >= 1)
)
write_csv_by_year(o3_site_day_out, "site_id",
                  file.path(clean_dir, "aqs_ozone_site_day_2006_2025.csv.gz"))

The site list is built from the clean site-day files, so every site in it has at least one clean day.

# First and last day with a clean value, for each pollutant.
pm25_span <- pm25_site_day_out |>
  summarize(pm25_first = min(date), pm25_last = max(date), .by = site_id)
o3_span <- o3_site_day_out |>
  summarize(o3_first = min(date), o3_last = max(date), .by = site_id)

# inner_join() keeps only sites with at least one clean day of either
# pollutant: a site whose every row was dropped above has nothing to offer.
aqs_sites <- sites_geo |>
  filter(!is.na(GEOID)) |>
  inner_join(full_join(pm25_span, o3_span, by = "site_id"), by = "site_id") |>
  transmute(
    site_id, latitude, longitude, GEOID, local_site_name,
    measures_pm25 = !is.na(pm25_first),
    measures_o3 = !is.na(o3_first),
    first_date = pmin(pm25_first, o3_first, na.rm = TRUE),
    last_date = pmax(pm25_last, o3_last, na.rm = TRUE)
  ) |>
  arrange(site_id)

# Grain: one row per site; every site is in a county of the frame.
stopifnot(
  !anyDuplicated(aqs_sites$site_id),
  all(aqs_sites$GEOID %in% county_ids),
  all(aqs_sites$measures_pm25 | aqs_sites$measures_o3),
  all(aqs_sites$first_date <= aqs_sites$last_date),
  all(between(aqs_sites$latitude, 24, 50)), all(between(aqs_sites$longitude, -125, -66))
)
write_csv(aqs_sites, file.path(clean_dir, "aqs_sites.csv"), na = "")
count(aqs_sites, measures_pm25, measures_o3)

Average the sites in each county and day, one year at a time, and write the county-day file. After that, the two site-day tables are removed from memory.

# County value = simple mean across the county's sites that reported that
# day. Counties with no monitor that day have no row. We build one year at a
# time: summarize() over all 5.4 million county-days at once needs about
# 2.5 GB of extra memory; year by year, the whole step stays under 1 GB.
# between() on dates is much faster than computing year(date) for millions
# of rows.
county_day_one_year <- function(y) {
  first_day <- make_date(y, 1, 1)
  last_day <- make_date(y, 12, 31)
  pm25_county_day <- pm25_site_day_out |>
    filter(between(date, first_day, last_day)) |>
    summarize(pm25_mean = round(mean(pm25), 2), n_sites_pm25 = n(),
              .by = c(GEOID, date))
  o3_county_day <- o3_site_day_out |>
    filter(between(date, first_day, last_day)) |>
    summarize(o3_mda8_ppb = round(mean(o3_mda8_ppb), 2), n_sites_o3 = n(),
              .by = c(GEOID, date))
  # full_join() keeps a county-day if either pollutant was measured; the
  # other pollutant's columns are then NA (no monitor reading, not zero).
  full_join(pm25_county_day, o3_county_day, by = c("GEOID", "date"))
}
aqs_county_day <- map(2006:2025, county_day_one_year) |>
  list_rbind()

# Grain: one row per county and day with at least one monitor reading.
stopifnot(
  n_distinct(aqs_county_day$GEOID, aqs_county_day$date) == nrow(aqs_county_day),
  all(aqs_county_day$GEOID %in% county_ids),
  all(!is.na(aqs_county_day$pm25_mean) | !is.na(aqs_county_day$o3_mda8_ppb)),
  sum(aqs_county_day$n_sites_pm25, na.rm = TRUE) == nrow(pm25_site_day_out),
  sum(aqs_county_day$n_sites_o3, na.rm = TRUE) == nrow(o3_site_day_out)
)
write_csv_by_year(aqs_county_day, "GEOID",
                  file.path(clean_dir, "aqs_county_day_2006_2025.csv.gz"))
# The site-day tables are saved; the monthly step needs only aqs_county_day.
rm(pm25_site_day_out, o3_site_day_out)
invisible(gc())

Finally, summarize the county-day file by month: average levels, the number of days with a reading, and the number of days above the levels of the national standards.

# Monthly summaries of the county-day values. 35.4 µg/m³ and 70 ppb are the
# levels of the 24-hour PM2.5 standard (35, after rounding) and the 8-hour
# ozone standard. A daily value above them is not by itself a violation:
# the standards apply to 3-year statistics.
aqs_county_month <- aqs_county_day |>
  rename(pm25_day = pm25_mean, o3_day = o3_mda8_ppb) |>
  # floor_date() moves each date to the first day of its month. It is one
  # cheap step, where year() and month() would each convert 5.4 million dates.
  mutate(month_start = floor_date(date, "month")) |>
  summarize(
    pm25_mean = round(mean(pm25_day, na.rm = TRUE), 2),
    pm25_days = sum(!is.na(pm25_day)),
    pm25_days_gt35 = sum(pm25_day > 35.4, na.rm = TRUE),
    o3_mda8_mean_ppb = round(mean(o3_day, na.rm = TRUE), 2),
    o3_days = sum(!is.na(o3_day)),
    o3_days_gt70 = sum(o3_day > 70, na.rm = TRUE),
    .by = c(GEOID, month_start)
  ) |>
  mutate(
    year = year(month_start), month = month(month_start), .after = GEOID,
    # mean() of no values gives NaN; store "no reading" as NA instead.
    pm25_mean = if_else(pm25_days > 0, pm25_mean, NA),
    o3_mda8_mean_ppb = if_else(o3_days > 0, o3_mda8_mean_ppb, NA)
  ) |>
  select(-month_start) |>
  arrange(GEOID, year, month)

# Grain: one row per county and month with at least one monitor reading.
stopifnot(
  nrow(distinct(aqs_county_month, GEOID, year, month)) == nrow(aqs_county_month),
  all(aqs_county_month$pm25_days + aqs_county_month$o3_days > 0),
  all(aqs_county_month$pm25_days <= 31), all(aqs_county_month$o3_days <= 31),
  all(aqs_county_month$pm25_days_gt35 <= aqs_county_month$pm25_days),
  all(aqs_county_month$o3_days_gt70 <= aqs_county_month$o3_days)
)
write_csv(aqs_county_month,
          file.path(clean_dir, "aqs_county_month_2006_2025.csv"), na = "")

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
aqs_sites.csv	2,773	a site in the contiguous U.S. or DC with at least one clean PM2.5 or ozone day	0.24 MB
aqs_pm25_site_day_2006_2025.csv.gz	4,214,339	a site-day with a PM2.5 value	19.1 MB (165.3 MB unzipped)
aqs_ozone_site_day_2006_2025.csv.gz	7,497,771	a site-day with an ozone MDA8 value	27.9 MB (240.0 MB unzipped)
aqs_county_day_2006_2025.csv.gz	5,448,481	a county-day with at least one PM2.5 or ozone reading	26.3 MB (142.7 MB unzipped)
aqs_county_month_2006_2025.csv	202,737	a county-month with at least one PM2.5 or ozone reading	6.3 MB

The network changes over time. The table counts sites, counties, and site-days in the two site-day files:

Year	PM2.5 sites	PM2.5 counties	PM2.5 site-days	Ozone sites	Ozone counties	Ozone site-days
2006	976	603	125,266	1,249	751	349,167
2007	944	595	133,786	1,275	764	355,700
2008	923	585	129,382	1,279	768	356,642
2009	929	589	142,389	1,289	779	363,366
2010	923	594	153,108	1,316	793	371,794
2011	839	547	148,927	1,314	797	378,040
2012	846	544	156,607	1,318	788	381,019
2013	877	567	172,528	1,302	786	381,543
2014	930	593	183,738	1,297	789	381,778
2015	946	611	195,375	1,285	783	378,528
2016	942	604	206,775	1,275	782	382,055
2017	958	604	220,012	1,274	783	390,471
2018	973	618	237,360	1,269	783	390,036
2019	964	616	250,854	1,250	774	380,996
2020	943	607	255,992	1,243	764	383,592
2021	967	610	270,948	1,239	763	382,237
2022	967	613	283,157	1,232	766	374,493
2023	988	623	298,456	1,220	759	375,119
2024	1,008	630	320,922	1,234	763	373,327
2025	1,016	630	328,757	1,202	750	367,868

Clean variables

aqs_sites.csv: one row per site, sorted by site_id. No cell is missing except local_site_name (blank for 203 sites).

Variable	Type	Units	Description
site_id	chr	—	9-character AQS site ID: state (2) + county (3) + site number (4), from the raw State Code, County Code, and Site Num
latitude	dbl	degrees	Site latitude (raw Latitude, NAD83 or WGS84)
longitude	dbl	degrees	Site longitude (raw Longitude; negative = west)
GEOID	chr	—	5-character county FIPS code (state + county), 2020 boundaries; matches counties_2020.csv. The AQS county code for all 2,773 sites (see “Checks and cautions”)
local_site_name	chr	—	Site name from the raw Local Site Name
measures_pm25	lgl	—	TRUE if the site has at least one day in the PM2.5 site-day file
measures_o3	lgl	—	TRUE if the site has at least one day in the ozone site-day file
first_date	date	—	First day with a clean PM2.5 or ozone value
last_date	date	—	Last day with a clean PM2.5 or ozone value

aqs_pm25_site_day_2006_2025.csv.gz: one row per site and day with a PM2.5 value, sorted by year, then site_id, then date. No cell is missing. Days without a valid reading have no row.

Variable	Type	Units	Description
site_id	chr	—	9-character AQS site ID; links to aqs_sites.csv
GEOID	chr	—	5-character county FIPS code of the site (2020 boundaries)
date	date	—	Local calendar day (raw Date Local, local standard time)
pm25	dbl	µg/m³	Daily PM2.5, 2 decimals: the mean over the site’s POCs of each POC’s chosen daily value (24 HOUR, else 24-HR BLK AVG, else 1 HOUR with at least 75% of hours), using None and Included rows. Can be negative (down to −9.70)
n_poc	dbl	count	Number of POCs averaged into pm25 (1–6). A T640 with both original and corrected data counts as 2 (see “A T640 can count twice”)
pm25_negative	lgl	—	TRUE if pm25 is below 0

aqs_ozone_site_day_2006_2025.csv.gz: one row per site and day with an ozone value, sorted by year, then site_id, then date. No cell is missing. Many sites report only in the warm months (ozone season).

Variable	Type	Units	Description
site_id	chr	—	9-character AQS site ID; links to aqs_sites.csv
GEOID	chr	—	5-character county FIPS code of the site (2020 boundaries)
date	date	—	Local calendar day (raw Date Local, local standard time)
o3_mda8_ppb	dbl	ppb	Maximum daily 8-hour average ozone (MDA8): raw 1st Max Value of the Ozone 8-hour 2015 rows × 1,000, averaged over POCs, 2 decimals
n_poc	dbl	count	Number of POCs averaged (1, 2, or 4)

aqs_county_day_2006_2025.csv.gz: one row per county and day on which at least one monitor in the county reported PM2.5 or ozone, sorted by year, then GEOID, then date. Use arrange(GEOID, date) after reading if you want each county’s days together. A county-day with no row had no reading at all.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (2020 boundaries); matches counties_2020.csv
date	date	—	Local calendar day
pm25_mean	dbl	µg/m³	Simple mean of pm25 over the county’s sites that reported that day, 2 decimals. Blank (NA) if no PM2.5 site reported; this happens on 2,563,528 county-days with ozone only
n_sites_pm25	dbl	count	Number of sites in pm25_mean; blank if none
o3_mda8_ppb	dbl	ppb	Simple mean of o3_mda8_ppb over the county’s sites that reported that day, 2 decimals. Blank (NA) if no ozone site reported
n_sites_o3	dbl	count	Number of sites in o3_mda8_ppb; blank if none

aqs_county_month_2006_2025.csv: one row per county and month with at least one county-day, sorted by GEOID, year, and month. All values come from the county-day file.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (2020 boundaries)
year	dbl	—	Calendar year, 2006–2025
month	dbl	—	Calendar month, 1–12
pm25_mean	dbl	µg/m³	Mean of pm25_mean over the county-days with a PM2.5 reading, 2 decimals. Blank (NA) if pm25_days is 0
pm25_days	dbl	days	Number of days with a PM2.5 reading (0–31)
pm25_days_gt35	dbl	days	Days with pm25_mean above 35.4 µg/m³, the highest value that still rounds to the 24-hour standard of 35 µg/m³
o3_mda8_mean_ppb	dbl	ppb	Mean of o3_mda8_ppb over the county-days with an ozone reading, 2 decimals. Blank (NA) if o3_days is 0
o3_days	dbl	days	Number of days with an ozone reading (0–31)
o3_days_gt70	dbl	days	Days with o3_mda8_ppb above 70 ppb, the level of the 2015 ozone standard

Checks and cautions

• Matches the project build. R/check_m03.R compares every clean file with the research project’s files (series “events included”). The site list has the same 2,773 sites, with the same GEOIDs and coordinates. The PM2.5 site-day file has the same 4,214,339 site-days; pm25 differs by at most 0.005 µg/m³ (rounding), and n_poc, GEOID, and pm25_negative are identical. The county-day file has the same 5,448,481 county-days and the same site counts; means differ by at most 0.0092 µg/m³ and 0.005 ppb. The county-month file has the same 202,737 county-months; all day counts agree except pm25_days_gt35, which is one day lower in 9 county-months, because we compare the rounded county-day values with 35.4. The project build does not store an ozone site-day file, but its county-day ozone values match ours.
• Raw files are complete. All 40 zips are byte-identical to the project’s copies, and every read has exactly the number of rows EPA lists.
• Sparse coverage. Only 1,080 of 3,108 counties have a site (PM2.5: 804, ozone: 897). On a typical day, 452 counties have a PM2.5 reading and 725 an ozone reading (medians over days). Many filter samplers run only every third or sixth day, and many ozone monitors run only in the warm months: 10.1% of ozone site-days fall in July but 6.0% in January. A county with no row on a day was not measured; do not treat it as clean air.
• “Included” is mostly T640 data in 2018–2023. EPA flags exceptional events, such as wildfire smoke, but the big rise of Included PM2.5 rows after 2017 is not smoke. In the raw 2018–2023 PM2.5 files (all rows, including those outside the contiguous U.S.), 99.6% of the original T640 rows (method codes 236 and 238) are Included, against 3.3% of other monitors’ rows (83,558 of 2,549,528) and 7.0% of the corrected T640 rows (736 and 738). So the Included count per year tells you little about smoke.
• Events excluded instead? The project also builds an “events excluded” series from the None + Excluded rows. It has 4,053,533 site-days: 160,806 of our site-days have no value in it, and on another 457,388 the value differs by more than 0.005 µg/m³. Most of these differences fall in 2018–2023 and come from the T640 flags described above, not from smoke. In June 2023, 9,864 of 24,028 site-days differ (mean 18.09 µg/m³ with events included, 17.40 with events excluded) and 1,404 are missing from the excluded series. To build that series yourself, change the filter in pm25_site_day() to event_type %in% c("None", "Excluded"). Be aware of what it does: most Included rows have no Excluded twin (EPA writes an Excluded row only for some flags), so this filter drops those days instead of giving them a smoke-free value. That is why 160,806 site-days are missing from that series.
• A T640 can count twice. EPA corrected the Teledyne T640 and T640X PM2.5 data in 2024 and kept both versions under the same parameter code, as separate POCs. Its data advisory recommends using the corrected data (736, 738) and excluding the originals (236, 238). Like the project build, our files keep both, so on 450,979 site-days (10.7%) one T640 enters the site mean twice, once original and once corrected. This happens mostly in 2017–2023: on 39.7% of site-days in 2022. At Queens College on 6 June 2023, pm25 = 86.37 µg/m³ is the mean of a filter (58.6), the original T640 (101.0), and the corrected T640 (99.5); following the advisory gives (58.6 + 99.5) / 2 = 79.05. Over all 450,979 site-days, dropping the originals lowers the site mean by 0.53 µg/m³ on average (median −0.50, or −8.1%; 90% of changes between −0.90 and −0.20). Another 6,125 site-days have an original T640 POC with no corrected twin. To follow the advisory, add filter(!method %in% c("236", "238")) before the duration ranking in pm25_site_day().
• Which PM2.5 row is used. Of the kept instrument-days, 36.5% use a 24 HOUR filter sample and 63.5% use EPA’s 24-HR BLK AVG. The script cannot check the completeness of the 24-HR BLK AVG values itself: their Observation Percent is always 100, because EPA writes them only for days that meet its completeness rule. The 1 HOUR fallback is never used: every instrument-day whose hourly values pass the 75% rule also has a higher-ranked row. The rule still matters. It removes 64,552 None or Included 1 HOUR rows with fewer than 18 hourly values. Only 76 of them have a higher-ranked row for the same instrument-day, so without the rule 44,288 extra site-days in the contiguous U.S. would enter through the fallback, each built from an incomplete day. In 2006–2007 the files have no continuous monitors, only filter samples.
• Negative values. 14,491 PM2.5 site-days (0.34%) are below 0, the lowest at −9.70 µg/m³; they are kept and flagged with pm25_negative. 5 ozone site-days are below 0 (lowest −3.00 ppb). Two county-day cells and one county-month cell are written as -0 (a mean that rounds to zero from below); read_csv() reads them as 0.
• Ozone completeness. For the 2015 standard, EPA counts an MDA8 day as complete when at least 13 of its 17 8-hour periods are available. 162,531 of the kept ozone rows (2.1%) have fewer; 1,345 of those are above 70 ppb. Like the project build, we keep them.
• County assignment. All 2,773 sites keep their AQS county code. These files give Connecticut sites the 2020 county codes, so no site in the contiguous U.S. needs point-in-polygon. For 8 sites the coordinates fall in a different county than the code, for example a DC school (110010025) that plots in Charles County, Maryland; we trust the code, as the project does (its planar test flags 6 of these 8). 76 sites are outside the contiguous U.S.: Alaska 18, Hawaii 22, Puerto Rico 17, Virgin Islands 3, Mexico 14, Canada 2.
• June 2023. County-days above 35.4 µg/m³ peak in June 2023 (862 of the year’s 1,747). On 7 June 2023 the five highest county means were all in Pennsylvania: Northampton (42095, 258.00 µg/m³), Lehigh (42077, 242.80), Lackawanna (42069, 216.75), Berks (42011, 211.85), and Montgomery (42091, 208.65). Philadelphia (42101, 207.71, mean of 6 sites) and Queens County, New York (36081, 202.60) came next.
• Extremes. The highest PM2.5 site-day is 833.80 µg/m³ (Lynn Water Treatment Plant, Essex County, Massachusetts, 10 November 2024). Lee Vining in Mono County, California, recorded 616–823 µg/m³ on 14–17 September 2020. The highest ozone MDA8 is 183 ppb.
• Local standard time. date is the local standard day at the monitor. Products built on UTC (Coordinated Universal Time, the clock at Greenwich without daylight saving) can be off by one day around midnight; keep this in mind when you join AQS with satellite or model data.
• Reading the files. read_csv() guesses GEOID and site_id as text and the counts as numbers, but give col_types anyway, with GEOID and site_id as character. The county files have long runs of blank cells, and a column that is blank in the first 1,000 rows would be guessed wrongly.
• Memory and time. The script took 4–6.5 minutes in our runs, most of it reading the 40 zips and building the county-day file. Its peak memory, as Activity Monitor counts it, was about 2.1 GB. The operating system’s maximum resident-set count was about 2.75 GB, because it also counts parts of the unzipped CSV files that readr keeps on disk and maps into memory. Writing the big files one year at a time and removing objects early with rm() brought the peak down from about 2.85 GB. On an 8 GB laptop, close other memory-heavy programs (browsers with many tabs, other R sessions) before you run it.

NOAA nClimGrid-Daily county weather

nClimGrid-Daily gives the daily maximum temperature, minimum temperature, and precipitation of every county in the contiguous United States. NOAA’s National Centers for Environmental Information (NCEI) builds it from daily reports of thousands of weather stations (the Global Historical Climatology Network–Daily, GHCNd). NCEI interpolates the reports (estimates values between the stations) onto a grid of cells about 4 km wide, scales the daily values to match its monthly nClimGrid product, and then averages the grid cells inside each county.

In this project, weather is a control variable: heat and rain move with smoke and also matter on their own, so a smoke effect estimated without weather can pick up a weather effect. The data tell you how warm and wet a county was on each day. They say nothing about wind, humidity, or smoke, and a single county-day value is an estimate with real error.

Provider	NOAA National Centers for Environmental Information (NCEI), nClimGrid-Daily version 1.0.0, county area averages
Coverage	2006–2025 (the provider covers 1951 to the present) · 3,107 county series covering the contiguous U.S. and DC; the clean files cover all 3,108 counties of counties_2020.csv
Raw unit	one row = one county in one month for one variable (tmax, tmin, or prcp), with 31 day columns
Clean unit	one row = one county on one day (daily files), or one county in one month (monthly file)
Licence and citation	U.S. Government work, public domain. Cite Durre, I., M. F. Squires, R. S. Vose, A. Arguez, W. S. Gross, J. R. Rennie, and C. J. Schreck (2022), “NOAA’s nClimGrid-Daily Version 1 – Daily gridded temperature and precipitation for the Contiguous United States since 1951,” NOAA NCEI, doi:10.25921/c4gt-r169, and Durre, I., A. Arguez, C. J. Schreck III, M. F. Squires, and R. S. Vose (2022), “Daily high-resolution temperature and precipitation fields for the Contiguous United States from 1951 to present,” Journal of Atmospheric and Oceanic Technology, doi:10.1175/JTECH-D-22-0024.1
Documentation	nClimGrid-Daily product page; README of the data folder and user guide (copies in wildfire_data/raw/nclimgrid/doc/); data folder

Raw files

File (in wildfire_data/raw/)	Size	Format	One row is
nclimgrid/cty/<year>/tmax-<YYYYMM>-cty-scaled.csv (240 files: one per month, 2006–2025, in one folder per year)	1.09 MB each; 262.4 MB in all	comma-separated text, ASCII, no header row; 37 columns, values padded to fixed width	one county in one month: daily maximum temperature (3,107 rows per file)
nclimgrid/cty/<year>/tmin-<YYYYMM>-cty-scaled.csv (240 files)	1.09 MB each; 262.4 MB in all	same	one county in one month: daily minimum temperature
nclimgrid/cty/<year>/prcp-<YYYYMM>-cty-scaled.csv (240 files)	1.09 MB each; 262.4 MB in all	same	one county in one month: daily precipitation
nclimgrid/doc/us-state-codes_ncei-to-fips.csv	0.001 MB	comma-separated text with a header row	one state: its NCEI state number and its FIPS code, the federal state number (48 rows)
nclimgrid/doc/nclimgrid-daily_v1-0-0_readme-web.txt	0.007 MB	plain text	provider README: file names, region types, and the CSV layout
nclimgrid/doc/nclimgrid-daily_v1-0-0_user-guide.pdf	0.12 MB	PDF	provider user guide: method, accuracy, references
nclimgrid/doc/nclimgrid-daily_status-web.txt	0.002 MB	plain text	provider release notes for version 1.0.0

In the file names, scaled means final values that NCEI has scaled to match its monthly product. The newest months appear first as prelim (preliminary) files; every file shipped here is scaled. NCEI also publishes tavg files and other region types (states, climate divisions, census tracts); the bundle has only the county files for tmax, tmin, and prcp.

Raw variables

The 720 county files (tmax-, tmin-, and prcp-<YYYYMM>-cty-scaled.csv) share this layout, described in the provider’s README and user guide; the files have no header row, so the names in parentheses are the ones the script gives the columns.

Variable	Type	Description
Column 1 (region_type)	text, 3 characters	Type of region; always cty (county) in these files
Column 2 (region_code)	text, 5 digits	NCEI region code: the 2-digit NCEI state number, then the 3-digit county FIPS code. It is not a FIPS county code: Monroe County, New York, is 30055 here (NCEI’s New York is 30) but 36055 in Census files. DC is 18511, filed under NCEI’s Maryland (18). The provider documents the state part; we confirmed the county part by matching all 3,106 codes other than DC to Census counties (DC’s 18511 is recoded to 11001 by hand)
Column 3 (region_name)	text	State postal abbreviation, a colon, and the county name, e.g., NY: Monroe County, VA: Rockbridge County, DC: District of Columbia
Column 4 (year)	integer, 4 digits	Year
Column 5 (month)	integer, 2 digits (01–12)	Month
Column 6 (element)	text, 4 characters	Variable in the file: TMAX daily maximum temperature (°C), TMIN daily minimum temperature (°C), PRCP daily precipitation (mm)
Columns 7–37 (day_1 … day_31)	number with 2 decimals, fixed width: 9 characters between the commas (a space, then the provider’s %8.2f format, e.g., 30.37); read_csv() trims the spaces	Value on day 1, 2, …, 31 of the month: the average over the county of the grid-cell values, in °C (TMAX, TMIN) or mm (PRCP). -999.99 marks a day that does not exist, such as February 30 or June 31. There are no other missing values in 2006–2025

The crosswalk us-state-codes_ncei-to-fips.csv translates NCEI state numbers into FIPS state codes.

Variable	Type	Description
state_name	text	State name without spaces, e.g., NewYork. The names of Illinois and Indiana are swapped: the row Indiana,IL,11,17 is Illinois. The codes in each row are correct
postal_code	text, 2 letters	U.S. Postal Service state abbreviation
NCEI_code	text, 2 digits	NCEI state number, 01–48: the 48 contiguous states in alphabetical order. NCEI uses it in station identifiers and in the region codes above
FIPS_code	text, 2 digits	FIPS state code (Federal Information Processing Series: the federal number of the state)

The README, status file, and user guide are text documents with no variables.

Cleaning steps

The script R/m04_nclimgrid.R makes these decisions:

1. Name the columns ourselves. The raw files have no header row. region_code is read as text, because codes such as 01001 start with a zero.
2. Drop days that do not exist. Each row has 31 day columns, whatever the month’s length. -999.99 is read as missing, and day columns past the end of the month are dropped. A missing value on a real day stops the script (there are none in 2006–2025).
3. Reshape twice. pivot_longer() turns the 31 day columns into rows (one row per county per day), and pivot_wider() then puts tmax, tmin, and prcp side by side.
4. Translate NCEI codes into GEOIDs. The GEOID is the 5-character county FIPS code used by the Census Bureau and by every other clean file. We take the state from the postal abbreviation, the first two characters of region_name (looked up in us_states_2020.csv), and the county from the last three digits of region_code. DC becomes 11001. NCEI’s own crosswalk gives the same states, which the script checks.
5. Fill the one missing county. NCEI publishes no series for Lexington city, Virginia (51678), and does not say why; size alone does not explain it, because the even smaller Falls Church city (5.3 km²) has one. Lexington is an independent city of 6.5 km² surrounded by Rockbridge County (51163). As in the research project, Lexington gets Rockbridge’s values, and the TRUE/FALSE column weather_imputed marks these copied rows. (The project stores the donor’s GEOID instead; we use TRUE/FALSE because a column that is blank on almost every row makes read_csv() guess its type wrong.) Every clean panel is then complete.
6. Average temperature. tavg_c = (tmax_c + tmin_c) / 2, the definition NCEI uses for its own average temperature.
7. One year at a time. The script builds, checks, and writes one year of county-days (about 1.1 million rows) before it reads the next, so it never holds all 22.7 million county-days in memory at once.
8. Monthly summaries. Means and totals of the daily values, and counts of hot days (tmax ≥ 32 °C), frost days (tmin ≤ 0 °C), and days in four precipitation bins. The thresholds and bins are the research project’s.

The first chunk sets the folders and reads the county frame from module m00.

# The raw files sit in one folder per year, 36 files per year
# (3 variables x 12 months). The daily output gets its own folder with one
# file per year: all 20 years in one CSV would be about 1 GB.
ncg_dir <- file.path(raw_dir, "nclimgrid", "cty")
day_dir <- file.path(clean_dir, "nclimgrid_county_day")
dir.create(day_dir, recursive = TRUE, showWarnings = FALSE)
ncg_years <- 2006:2025

# The county frame from module m00: 3,108 counties (48 states + DC).
counties <- read_csv(file.path(clean_dir, "counties_2020.csv"),
                     col_types = cols(.default = col_character()),
                     col_select = c(GEOID, county_name, state_abbr))
stopifnot(nrow(counties) == 3108)

Next we read one raw file, June 2023 maximum temperature, with our own column names and types, and look at its wide layout.

# The files have no header row. Each row is one county in one month:
# 6 ID columns, then 31 day columns, whatever the length of the month.
# We name the columns ourselves. region_code stays text so that its
# leading zero survives ("01001", not 1001).
ncg_names <- c("region_type", "region_code", "region_name", "year", "month",
               "element", str_c("day_", 1:31))
ncg_types <- cols(
  region_type = col_character(), region_code = col_character(),
  region_name = col_character(), year = col_integer(), month = col_integer(),
  element = col_character(), .default = col_double()
)

# Daily maximum temperature in June 2023, the month Canadian smoke turned
# New York City's sky orange. -999.99 is NCEI's code for a day that does
# not exist (June 31), so we read it as NA.
tmax_jun23 <- read_csv(
  file.path(ncg_dir, "2023", "tmax-202306-cty-scaled.csv"),
  col_names = ncg_names, col_types = ncg_types, na = "-999.99"
)
tmax_jun23 |>
  select(region_code:element, day_1, day_2, day_30, day_31) |>
  head(3)

This is the step to understand well: pivot_longer() stacks the 31 day columns into two columns, day (which column the value came from) and tmax_c (the value). The file grows from 3,107 rows to 96,317 rows, and dropping June 31 leaves 93,210.

# This file is "wide": the days are spread across 31 columns. pivot_longer()
# stacks them into rows, one row per county per day. names_prefix drops
# "day_" from the column names, and names_transform turns "7" into 7L.
tmax_jun23_long <- tmax_jun23 |>
  pivot_longer(
    cols = starts_with("day_"),
    names_to = "day", names_prefix = "day_",
    names_transform = list(day = as.integer),
    values_to = "tmax_c"
  )
nrow(tmax_jun23_long)   # 3,107 counties x 31 day columns = 96,317 rows

# Keep the days that exist (June has 30), then build a real Date.
tmax_jun23_long <- tmax_jun23_long |>
  filter(day <= days_in_month(make_date(year, month, 1L))) |>
  mutate(date = make_date(year, month, day))

stopifnot(nrow(tmax_jun23_long) == 3107 * 30, !anyNA(tmax_jun23_long$tmax_c))
tmax_jun23_long |>
  select(region_code, region_name, date, tmax_c) |>
  head(3)

The county codes need a lookup table. We build it from the same file and check it against the county frame.

# region_code is NOT a FIPS code. Its first two digits are NCEI's own state
# number (01 Alabama ... 48 Wyoming, alphabetical, without Alaska and Hawaii);
# its last three digits are the county's FIPS code. The postal abbreviation,
# the first two characters of region_name ("AL: Autauga"), is easier to
# translate: the m00 state table maps it to the 2-digit FIPS state code.
states <- read_csv(file.path(clean_dir, "us_states_2020.csv"),
                   col_types = cols(.default = col_character()),
                   col_select = c(STATEFP, state_abbr))

# Every raw file lists the same 3,107 counties in the same order, so one
# file is enough to build the lookup table.
ncg_counties <- tmax_jun23 |>
  distinct(region_code, region_name) |>
  mutate(state_abbr = str_sub(region_name, 1, 2)) |>
  left_join(states, by = "state_abbr") |>
  mutate(
    GEOID = str_c(STATEFP, str_sub(region_code, 3, 5)),
    # DC is filed under county number 511; its Census code is 11001.
    GEOID = if_else(GEOID == "11511", "11001", GEOID)
  )

stopifnot(
  nrow(ncg_counties) == 3107, !anyNA(ncg_counties$GEOID),
  !anyDuplicated(ncg_counties$GEOID),
  all(ncg_counties$GEOID %in% counties$GEOID)
)
# The one frame county with no nClimGrid series: Lexington city, VA.
counties |> filter(!GEOID %in% ncg_counties$GEOID)

As a second check, we translate the codes with NCEI’s own crosswalk and list every disagreement, including any code the crosswalk lacks. Only DC disagrees.

# NCEI also ships a crosswalk from its state numbers to FIPS state codes.
# Using it on the first two digits of region_code gives the same state as
# the postal abbreviation for every county but one: DC, which NCEI numbers
# as county 511 of Maryland (NCEI 18 = FIPS 24). The file's state_name
# column swaps Illinois and Indiana, but its codes are correct.
ncei_states <- read_csv(
  file.path(raw_dir, "nclimgrid", "doc", "us-state-codes_ncei-to-fips.csv"),
  col_types = cols(.default = col_character())
)
# A code missing from the crosswalk gives FIPS_code = NA after the join.
# FIPS_code != STATEFP is then NA, and filter() silently drops NA rows,
# so we keep them on purpose with is.na(): they are disagreements too.
disagree <- ncg_counties |>
  mutate(NCEI_code = str_sub(region_code, 1, 2)) |>
  left_join(select(ncei_states, NCEI_code, FIPS_code), by = "NCEI_code") |>
  filter(is.na(FIPS_code) | FIPS_code != STATEFP)

disagree |> select(region_code, region_name, STATEFP, FIPS_code)
stopifnot(identical(disagree$region_code, "18511"))

Now we wrap the read-and-pivot steps in a function that works for any raw file. It returns a long table with one row per county, day, and variable.

# The same read-and-pivot steps as above, for any of the 720 files.
# element is lowercased ("TMAX" -> "tmax") so it can become a column name.
read_ncg_file <- function(path) {
  out <- read_csv(path, col_names = ncg_names, col_types = ncg_types,
                  na = "-999.99") |>
    pivot_longer(
      cols = starts_with("day_"),
      names_to = "day", names_prefix = "day_",
      names_transform = list(day = as.integer),
      values_to = "value"
    ) |>
    filter(day <= days_in_month(make_date(year, month, 1L))) |>
    transmute(region_code, element = str_to_lower(element),
              date = make_date(year, month, day), value)
  # -999.99 should only fill days that a month does not have. NA on a real
  # day would be a gap in the data, so stop and look. Every county code
  # must also be in the lookup table built from June 2023.
  stopifnot(!anyNA(out$value), all(out$region_code %in% ncg_counties$region_code))
  out
}

One month is three files. After stacking them, pivot_wider() spreads the element column back out into tmax, tmin, and prcp columns.

# One month is three files: tmax, tmin, and prcp. Stack them, then put the
# three variables side by side with pivot_wider(), the reverse of
# pivot_longer(): the values of element become the new column names.
# Result: one row per county per day. (%02d pads the month: 6 -> "06".)
read_ncg_month <- function(yr, mo) {
  files <- file.path(ncg_dir, yr, sprintf("%s-%d%02d-cty-scaled.csv",
                                          c("tmax", "tmin", "prcp"), yr, mo))
  stopifnot(all(file.exists(files)))
  map(files, read_ncg_file) |>
    list_rbind() |>
    pivot_wider(names_from = element, values_from = value)
}

read_ncg_month(2023, 6) |> head(3)

A year is twelve months. This function also adds GEOIDs, fills Lexington city, computes tavg_c, and checks that the panel is complete. We call the data frame ncg_day, not day, because day is already the name of a column made by pivot_longer() and of a lubridate function.

# One year: read 12 months and swap NCEI codes for GEOIDs. NCEI publishes
# no series for Lexington city, VA (51678, 6.5 km2), and the provider does
# not say why (even smaller Falls Church city, 51610, has one). Like the
# research project, we copy the values of Rockbridge County (51163), which
# surrounds it. weather_imputed is TRUE on these copied rows and FALSE on
# every other row. It is never blank, so read_csv() always reads it as
# logical, even without col_types.
build_ncg_year <- function(yr) {
  ncg_day <- map(1:12, \(mo) read_ncg_month(yr, mo)) |>
    list_rbind() |>
    left_join(select(ncg_counties, region_code, GEOID), by = "region_code") |>
    transmute(GEOID, date, tmax_c = tmax, tmin_c = tmin, prcp_mm = prcp,
              weather_imputed = FALSE)

  lexington <- ncg_day |>
    filter(GEOID == "51163") |>
    mutate(GEOID = "51678", weather_imputed = TRUE)

  ncg_day <- bind_rows(ncg_day, lexington) |>
    # (tmax + tmin) / 2 has at most 3 decimals, so keep all 3.
    mutate(tavg_c = round((tmax_c + tmin_c) / 2, 3), .after = tmin_c) |>
    arrange(GEOID, date)

  # Complete panel: every frame county on every day of the year, once.
  n_dates <- as.integer(make_date(yr, 12L, 31L) - make_date(yr, 1L, 1L)) + 1L
  stopifnot(
    nrow(ncg_day) == 3108 * n_dates,
    setequal(ncg_day$GEOID, counties$GEOID),
    nrow(distinct(ncg_day, GEOID, date)) == nrow(ncg_day),
    !anyNA(ncg_day),
    sum(ncg_day$weather_imputed) == n_dates,
    all(ncg_day$tmin_c <= ncg_day$tmax_c), all(ncg_day$prcp_mm >= 0)
  )
  ncg_day
}

Before running twenty years, try one. The output shows Monroe County, New York (Rochester), around the smoke days of June 6–8, 2023: no rain and highs near 20 °C.

# Try one year before running all twenty. 2023 has 365 days, so the
# result should have 3,108 x 365 = 1,134,420 rows.
day_2023 <- build_ncg_year(2023)
glimpse(day_2023)

# Monroe County, NY (Rochester) during the June 2023 smoke days.
day_2023 |>
  filter(GEOID == "36055", between(date, ymd("2023-06-05"), ymd("2023-06-09")))
day_2023 |> count(weather_imputed)

The monthly summary uses summarise() with .by, so it needs no group_by(). sum() of a TRUE/FALSE condition counts the days that meet it.

# Monthly summaries of one year of county-days. The thresholds follow the
# research project: hot day = tmax >= 32 C (about 90 F), frost day =
# tmin <= 0 C, and four precipitation bins (dry, light, moderate, heavy).
summarise_ncg_month <- function(ncg_day) {
  ncg_day |>
    mutate(year = as.integer(year(date)), month = as.integer(month(date))) |>
    summarise(
      tmax_mean_c = round(mean(tmax_c), 2),
      tmin_mean_c = round(mean(tmin_c), 2),
      tavg_mean_c = round(mean(tavg_c), 2),
      prcp_total_mm = round(sum(prcp_mm), 2),
      days_tmax_ge32 = sum(tmax_c >= 32),
      days_tmin_le0 = sum(tmin_c <= 0),
      days_prcp_0 = sum(prcp_mm == 0),
      days_prcp_0_5 = sum(prcp_mm > 0 & prcp_mm <= 5),
      days_prcp_5_15 = sum(prcp_mm > 5 & prcp_mm <= 15),
      days_prcp_gt15 = sum(prcp_mm > 15),
      n_days = n(),
      weather_imputed = any(weather_imputed),
      .by = c(GEOID, year, month)
    )
}

summarise_ncg_month(day_2023) |> filter(GEOID == "36055", month %in% 6:7)

The loop runs all twenty years and writes one compressed CSV per year. The whole script takes about 3 to 7 minutes and 1.0 to 1.6 GB of memory on a shared 16 GB Mac.

# Build, check, and write each year, then keep only its monthly summary.
# Each new year replaces the last one in `ncg_day`, so the 22.7 million
# county-days are never in memory all at once. This takes a few minutes.
# Grain of each file: one row per county x day (GEOID + date unique),
# 3,108 counties x 365 or 366 days.

# day_2023 was a trial run and the loop builds 2023 again, so free its
# memory first (if you skipped the trial, there is nothing to free).
if (exists("day_2023")) rm(day_2023)
ncg_month_list <- vector("list", length(ncg_years))
for (i in seq_along(ncg_years)) {
  yr <- ncg_years[i]
  ncg_day <- build_ncg_year(yr)
  stopifnot(nrow(ncg_day) == 3108 * if_else(leap_year(yr), 366, 365))
  write_csv(ncg_day, file.path(day_dir, str_glue("nclimgrid_county_day_{yr}.csv.gz")),
            na = "")
  ncg_month_list[[i]] <- summarise_ncg_month(ncg_day)
  message(yr, ": ", nrow(ncg_day), " county-days written")
}
ncg_month <- list_rbind(ncg_month_list) |> arrange(GEOID, year, month)

The monthly file is checked and written once all years are done.

# Grain: one row per county x month, 3,108 counties x 240 months = 745,920
# rows, with GEOID + year + month unique. Each month's day counts must add
# up to the number of days in the month.
stopifnot(
  nrow(ncg_month) == 3108 * 240,
  nrow(distinct(ncg_month, GEOID, year, month)) == nrow(ncg_month),
  !anyNA(ncg_month),
  all(ncg_month$n_days == days_in_month(make_date(ncg_month$year, ncg_month$month, 1L))),
  with(ncg_month, all(days_prcp_0 + days_prcp_0_5 + days_prcp_5_15 + days_prcp_gt15 == n_days)),
  all(ncg_month$tmin_mean_c <= ncg_month$tmax_mean_c),
  all(between(ncg_month$days_tmax_ge32, 0, ncg_month$n_days)),
  sum(ncg_month$weather_imputed) == 240
)
write_csv(ncg_month, file.path(clean_dir, "nclimgrid_county_month_2006_2025.csv"),
          na = "")

Finally, read the files back the way later modules will: with plain read_csv() and no col_types. The checks confirm that every column gets the right type.

# Read the clean files the way later modules will, with plain read_csv().
# Every file is sorted by GEOID and starts with "01001", so read_csv()
# sees the leading zero and keeps GEOID as text. weather_imputed is
# TRUE/FALSE on every row, so it is read as logical. Giving
# col_types = cols(GEOID = col_character()) is still a good habit.
day_check <- read_csv(file.path(day_dir, "nclimgrid_county_day_2023.csv.gz"),
                      show_col_types = FALSE)
month_check <- read_csv(file.path(clean_dir, "nclimgrid_county_month_2006_2025.csv"),
                        show_col_types = FALSE)
stopifnot(
  is.character(day_check$GEOID), is.Date(day_check$date),
  is.logical(day_check$weather_imputed), nrow(problems(day_check)) == 0,
  nrow(day_check) == 3108 * 365, sum(day_check$weather_imputed) == 365,
  is.character(month_check$GEOID), is.logical(month_check$weather_imputed),
  nrow(problems(month_check)) == 0,
  nrow(month_check) == 3108 * 240, sum(month_check$weather_imputed) == 240
)

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
nclimgrid_county_day/nclimgrid_county_day_<year>.csv.gz (20 files, 2006–2025)	1,134,420 per year (1,137,528 in the leap years 2008, 2012, 2016, 2020, 2024); 22,703,940 in all	one county on one day	12.4–12.6 MB each; 250.3 MB in all
nclimgrid_county_month_2006_2025.csv	745,920	one county in one month (3,108 counties × 240 months)	44.0 MB

Clean variables

The 20 daily files share these columns. Rows are sorted by GEOID and date. No cell is missing. The types are the ones plain read_csv() gives.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (2-digit state + 3-digit county), as in counties_2020.csv
date	date	—	Day, YYYY-MM-DD, as dated by NCEI. Dates are local dates at the stations, not UTC (Coordinated Universal Time) dates, and a day is not a strict midnight-to-midnight day (see the cautions)
tmax_c	dbl	°C	Daily maximum temperature, county average of grid cells (2 decimals)
tmin_c	dbl	°C	Daily minimum temperature, county average of grid cells (2 decimals)
tavg_c	dbl	°C	(tmax_c + tmin_c) / 2, kept at 3 decimals so it is exact
prcp_mm	dbl	mm	Daily precipitation, county average of grid cells (2 decimals). 0 means the county average rounds to 0.00 mm (below 0.005 mm): usually a dry day across the county, but a few grid cells may have had a trace
weather_imputed	lgl	—	TRUE on the rows of Lexington city, Virginia (51678): its values are copied from Rockbridge County (51163), which surrounds it. FALSE on every other row, where the values are the county’s own series

nclimgrid_county_month_2006_2025.csv: one row per county and month, sorted by GEOID, year, and month. No cell is missing. The types are the ones plain read_csv() gives.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code
year	dbl	—	Year, 2006–2025
month	dbl	—	Month, 1–12
tmax_mean_c	dbl	°C	Mean of the month’s daily tmax_c (2 decimals)
tmin_mean_c	dbl	°C	Mean of the month’s daily tmin_c (2 decimals)
tavg_mean_c	dbl	°C	Mean of the month’s daily tavg_c (2 decimals)
prcp_total_mm	dbl	mm	Total precipitation in the month: the sum of daily prcp_mm (2 decimals)
days_tmax_ge32	dbl	days	Hot days: days with tmax_c ≥ 32 °C (89.6 °F)
days_tmin_le0	dbl	days	Frost days: days with tmin_c ≤ 0 °C
days_prcp_0	dbl	days	Dry days: prcp_mm = 0, a county average that rounds to 0.00 mm (a few grid cells may still have had a trace)
days_prcp_0_5	dbl	days	Days with 0 < prcp_mm ≤ 5
days_prcp_5_15	dbl	days	Days with 5 < prcp_mm ≤ 15
days_prcp_gt15	dbl	days	Days with prcp_mm > 15
n_days	dbl	days	Days in the month (28–31). Every day is present, so the four precipitation bins add up to n_days
weather_imputed	lgl	—	TRUE on the 240 rows of Lexington city (51678), whose values are copied from Rockbridge County (51163); FALSE otherwise, as in the daily files

Checks and cautions

• Raw copies are exact. All 724 shipped files (720 county CSVs and 4 documents) are byte-identical (same MD5 checksum) to the research project’s downloads of 2026-09-27. Each county file has 3,107 rows and 37 columns, and all 720 files list the same counties in the same order. In 2006–2025, -999.99 appears only on days that do not exist, and every file is a final scaled file.
• Daily files match the research project. For all 20 years, the daily files have the same 22,703,940 county-days as the project’s nclimgrid_county_day panel. tmax_c, tmin_c, and prcp_mm are identical (largest difference 0), tavg_c agrees to within 10⁻¹⁴ °C, and weather_imputed is TRUE on exactly the rows where the project records a donor.
• Monthly file matches the research project. All 745,920 rows match the project’s nclimgrid_county_month.parquet for 2006–2025. Hot-day, frost-day, precipitation-bin, and n_days counts are identical; monthly means differ by at most 0.005 °C (our rounding to 2 decimals), and precipitation totals are identical. As a further check, recomputing the 2023 monthly totals and hot-day counts from the 2023 daily file gives the same values.
• The panels are complete. Every county in counties_2020.csv has a row for every day and every month of 2006–2025, and no cell is empty. Because no county-day or county-month is left out, there are no absent rows to interpret.
• Lexington city is a copy. Its 7,305 daily rows and 240 monthly rows are Rockbridge County’s values. Keep them if you need a complete panel; drop rows with weather_imputed == TRUE if an analysis should use only real county series. Rockbridge is 1,545 km² and Lexington 6.5 km², so the copy describes the surrounding countryside more than the town.
• Never join region_code to GEOID directly. 1,591 of the 3,107 NCEI codes are also valid GEOIDs, but only Alabama’s 67 are the right county, because Alabama is the one state whose NCEI number (01) equals its FIPS code. For example, 30055 is Monroe County, New York, in nClimGrid and McCone County, Montana, in Census files. A direct join would give 1,524 counties another county’s weather without any error message.
• County values are area averages. Each value is the mean of the grid cells inside the county, weighted by the cosine of latitude (a grid cell covers less ground farther north). It is not weighted by population, so in large western counties it mostly describes empty land, not the towns where people live.
• Single days are uncertain. NCEI’s tests for 1991–2020 found that half of the stations had an average interpolation error of 1 °C or less for tmax and tmin, and a median error of 2 mm or less for precipitation. Errors are largest in the mountains of the West and on days with heavy rain, and the smoothing gives precipitation a slight dry bias. NCEI discourages relying on single-day values for single places; averages over months or over many counties are more reliable.
• What a “day” is. NCEI builds each day by interpolating station reports taken in the morning or at midnight, so a day is not a strict midnight-to-midnight local day. The provider lists differences in observation time between neighboring stations as one source of error. Dates are local dates at the stations, not UTC dates. Data dated in UTC, such as the ERA5 days of module m09, cover different hours, so keep this in mind when you line up weather and smoke on specific days.
• Smoke can change the weather. Thick smoke blocks sunlight and can cool afternoon temperatures, so on heavy smoke days tmax_c is partly a result of the smoke. Same-day tmax_c can therefore be a poor control in a model of smoke effects.
• Thresholds are the project’s choices. A hot day is tmax_c ≥ 32 °C (about 90 °F) and a frost day is tmin_c ≤ 0 °C. 32.8% of county-months have at least one hot day and 51.7% at least one frost day. You can compute other thresholds (for example, 35 °C) from the daily files with the same sum() pattern.
• Reading the files. Plain read_csv() with no col_types reads every column correctly: GEOID as text (each file starts with 01001, so the leading zero is seen), date as a date, weather_imputed as logical, and every number, including year, month, and the day counts, as dbl. Chunk m04-read-back checks this. Adding col_types = cols(GEOID = col_character()) is still a good habit when you read only part of a file or filter it first.
• More monthly measures in the project. The research project’s monthly weather file also counts days in 5 °C bins of daily mean temperature (below −10 °C, −10 to −5 °C, …, 30 °C and above) and keeps the within-month standard deviation of tavg and the month’s highest tmax. They are not in this file; you can compute them from the daily files with cut() and summarise().
• Years not in the bundle. NCEI covers 1951 to the present. The research project also uses 1991–2005 and January–March 2026; those files are not in the student bundle.
• Memory and time. The whole script runs in about 3 to 7 minutes and uses 1.0 to 1.6 GB of memory on a shared 16 GB Mac (our latest run: 296 seconds, 1.4 GB), because it works on one year of county-days at a time.

ERA5 winds and surface weather

ERA5 is a reanalysis: a reconstruction of the weather for every hour since 1940, on a global grid of cells 0.25° of latitude and longitude wide (about 28 km north–south and 18–25 km east–west over the contiguous U.S.). The European Centre for Medium-Range Weather Forecasts (ECMWF) makes it for the Copernicus Climate Change Service (C3S). It feeds millions of observations from satellites, weather balloons, aircraft, ships, and weather stations into a weather-forecast model, so every cell has a value for every hour, even where nobody measured. The research project downloaded daily means of the hourly values over North America and averaged the surface weather and the height of the 500 hPa pressure level to counties, giving more weight to cells where more people live.

In this project ERA5 tells you which way and how hard the wind blew over a county each day (for example, the northerly winds over New York during the Quebec smoke of June 2023), how deep the layer of air that mixes near the ground was (a shallow layer keeps smoke close to the ground), how much sunlight reached the ground (smoke dims it), and the temperature and humidity that any smoke analysis has to account for. It does not measure smoke. A county’s daily-mean surface wind is also not the path the smoke took: smoke travels for days, mostly above the surface. ERA5 values are model estimates for grid cells, not readings at a weather station, and its days are UTC days, not local days.

Provider	European Centre for Medium-Range Weather Forecasts (ECMWF) for the Copernicus Climate Change Service (C3S); distributed by the Copernicus Climate Data Store (CDS). County averages: the research project (script scripts/R/02_winds.R), exported for this course by the instructor
Coverage	Raw (research project only): daily, January 2003 to March 2026 for surface fields and January 2003 to January 2026 for the 850/700 hPa winds; January 1991 to March 2026 for the 500 hPa geopotential · three rectangles over North America (see “Raw files”). Clean: daily and monthly, 2006–2025 · the 3,108 contiguous-U.S. counties (DC included)
Raw unit	one grid cell (0.25° × 0.25°) × UTC day (× pressure level for the B1 winds)
Clean unit	one row = a county × UTC day (instructor export, one file per year) or a county × month (built in this section)
Licence and citation	CC BY 4.0 (free to use and share if you credit the source). Cite the reanalysis: Hersbach, H., Bell, B., Berrisford, P., et al. (2020). The ERA5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730), 1999–2049. https://doi.org/10.1002/qj.3803. Cite the datasets from the Copernicus Climate Change Service (C3S) Climate Data Store (CDS): ERA5 post-processed daily statistics on pressure levels (https://doi.org/10.24381/cds.50314f4c) and on single levels (https://doi.org/10.24381/cds.4991cf48), and ERA5 hourly data on pressure levels (https://doi.org/10.24381/cds.bd0915c6) and on single levels (https://doi.org/10.24381/cds.adbb2d47)
Documentation	ERA5 data documentation; CDS pages for the daily statistics on single levels and on pressure levels; ECMWF parameter database (definitions of u, v, z, ssrd, blh, t2m, d2m, u10, v10); Hersbach et al. (2020)

Raw files

No raw ERA5 file ships in the bundle. The research project’s 979 monthly files take 17.1 GB, and downloading them requires a free Copernicus Climate Data Store account. They are listed here so that you know what the clean files are made from. The project requested three “bundles”, one NetCDF file per bundle and month (NetCDF-4 is a file format for labeled multi-dimensional arrays):

• B1 holds the winds at the 850 and 700 hPa pressure levels, about 1.5 km and 3 km above sea level, over 14–84° N and 180–50° W (most of North America and its oceans, where the smoke starts and travels).
• B2 holds the geopotential at 500 hPa, about 5.5 km up, over 20–75° N and 145–50° W.
• B3 holds six surface fields over 24–50° N and 125–66° W (the contiguous U.S.).

Every value is the mean of the 24 hourly values of one UTC day. UTC (Coordinated Universal Time) is the clock time at Greenwich, England. In New York a UTC day runs from 8 pm the evening before to 8 pm in summer (7 pm to 7 pm in winter); in California it runs from 5 pm to 5 pm in summer.

The files came by two routes with the same result. 67 of them came from the CDS “daily statistics” service as delivered: the B1 and B3 files for July 2023 (zip archives holding one NetCDF per variable) and 65 B2 files (January 1991 to April 1995, all of 2023, and April 2024). That service’s queue was too slow for the rest, so the project downloaded the other 912 months as hourly data and computed the daily means itself with the service’s rule (the same 24 hours and the same single-precision arithmetic). On the months it checked, the two routes gave identical values except on one day per month, where some cells differ by one or two of the smallest steps a single-precision number can resolve (at most 0.0008 gpm for the 500 hPa height and 0.00024 m for boundary-layer height).

File (in the research project’s data/raw/era5/; not shipped)	Size	Format	One row is
B1_pl_uv/era5_daymean_B1_<YYYY-MM>.nc, 277 monthly files, 2003-01 to 2026-01 (2023-07 is era5_daymean_B1_2023-07.zip)	39.5–53.2 MB each; 12.1 GB in all	NetCDF-4 (the zip holds one NetCDF per variable)	not rows but arrays: one value per grid cell × pressure level × day, 521 × 281 cells × 2 levels × 28–31 days
B2_pl_z500/era5_daymean_B2_<YYYY-MM>.nc, 423 monthly files, 1991-01 to 2026-03	4.3–4.9 MB each; 1.97 GB in all	NetCDF-4	arrays: one value per grid cell × day, 381 × 221 cells × 28–31 days
B3_sl/era5_daymean_B3_<YYYY-MM>.nc, 279 monthly files, 2003-01 to 2026-03 (2023-07 is era5_daymean_B3_2023-07.zip)	10.2–11.4 MB each; 3.07 GB in all	NetCDF-4 (the zip holds one NetCDF per variable)	arrays: one value per grid cell × day for each of 6 fields, 237 × 105 cells × 28–31 days
_manifest_era5.csv	1.05 MB	CSV	one file record (1,025 rows: the 979 files in use, status downloaded, plus 46 earlier files marked superseded): CDS request, download date, size, checksums, licence
README_era5.md	0.05 MB	Markdown text	the project’s download log (routes, requests, checks)

Raw variables

The tables describe the files as ncdf4 shows them, with dimensions in R’s order (tools that use C order, such as ncdump, list them reversed). Both routes give the same variable names, dimensions, units, and grids. They differ in two small ways. The CDS files store the number coordinate as a 64-bit integer and define the valid_time dimension first, while the project’s files store number as a 32-bit integer (each variable’s own dimension order is the same in both). And the attributes differ: the CDS files carry the provider’s GRIB_ metadata, and the files the project computed carry provenance attributes (hourly_window, cell_methods, era5_route, CDS job ids, and checksums). Every data variable is stored as a single-precision float (float32) with NaN as the missing-value code; no value is missing in the files checked (July 2024 for all three bundles, plus June and July 2023). Quoted names and definitions come from the ECMWF parameter database. The example ranges are from July 2024.

The first table covers the B1 files B1_pl_uv/era5_daymean_B1_<YYYY-MM>.nc.

Variable	Dimensions	Units	Description
longitude	longitude	degrees_east	Longitude of each cell center, −180 to −50 in steps of 0.25 (521 values; negative = west of Greenwich)
latitude	latitude	degrees_north	Latitude of each cell center, stored from 84 down to 14 in steps of 0.25 (281 values)
pressure_level	pressure_level	hPa	850 and 700. Air pressure falls with height, so a pressure level is a height in the air: average sea-level pressure is about 1,013 hPa (hectopascals), and the 850 and 700 hPa levels lie about 1.5 km and 3 km above sea level
valid_time	valid_time	days since <first day of the month> 00:00:00	The day: 0 is the first of the month. Each value is the mean over that UTC day. Calendar: proleptic Gregorian (the ordinary calendar)
u	longitude, latitude, pressure_level, valid_time	m s**-1 (m/s)	“U component of wind”: the eastward component of the wind, “the horizontal speed of air moving towards the east”; negative values mean air moving toward the west. Daily mean of the hourly values at 00–23 UTC. July 2024: −25.7 to 29.6 m/s
v	longitude, latitude, pressure_level, valid_time	m s**-1 (m/s)	“V component of wind”: the northward component of the wind; negative values mean air moving toward the south. Daily mean, as for u. July 2024: −24.9 to 22.9 m/s
number	none (a single value)	1	“Ensemble member numerical id”. Always 0: ERA5’s main reanalysis, not one of the extra runs ECMWF makes to measure uncertainty

The B2 files B2_pl_z500/era5_daymean_B2_<YYYY-MM>.nc have the same coordinate variables over a smaller area and one pressure level, and one data variable.

Variable	Dimensions	Units	Description
longitude	longitude	degrees_east	Cell centers from −145 to −50 in steps of 0.25 (381 values)
latitude	latitude	degrees_north	Cell centers from 75 down to 20 in steps of 0.25 (221 values)
pressure_level	pressure_level	hPa	500 (one value), about 5.5 km above sea level
valid_time	valid_time	days since <first day of the month> 00:00:00	The day, as in B1
z	longitude, latitude, pressure_level, valid_time	m**2 s**-2	“Geopotential”: “the gravitational potential energy of a unit mass, at a particular location, relative to mean sea level”. Dividing by g = 9.80665 m/s² gives the geopotential height in geopotential meters (gpm), close to the height above sea level, in meters, at which the pressure is 500 hPa. A high 500 hPa height is a ridge (often warm, dry, and calm under it); a low one is a trough. Daily mean of the hourly values at 00–23 UTC. July 2024: 50,480 to 59,140 m²/s² (5,147 to 6,031 gpm)
number	none	1	Ensemble member id, always 0, as in B1

The B3 files B3_sl/era5_daymean_B3_<YYYY-MM>.nc hold surface (“single-level”) fields, so they have no pressure_level dimension.

Variable	Dimensions	Units	Description
longitude	longitude	degrees_east	Cell centers from −125 to −66 in steps of 0.25 (237 values)
latitude	latitude	degrees_north	Cell centers from 50 down to 24 in steps of 0.25 (105 values)
valid_time	valid_time	days since <first day of the month> 00:00:00	The day, as in B1
ssrd	longitude, latitude, valid_time	J m**-2	“Surface short-wave (solar) radiation downwards”: “the amount of solar radiation … that reaches a horizontal plane at the surface”, direct and diffuse sunlight together. ERA5 stores it as the energy received during each hour, in joules per square meter. The file holds the daily mean of the 24 hourly amounts that cover 00–24 UTC (each hourly amount is labeled with the end of its hour, hence the time_shift attribute -1 days +23:00:00). Divide by 3,600 seconds to get the average power over the whole day, nights included, in W/m² (watts per square meter). July 2024: 73,970 to 1,401,000 J/m² (20.5 to 389 W/m²)
blh	longitude, latitude, valid_time	m	“Boundary layer height”: “the depth of air next to the Earth’s surface which is most affected by the resistance to the transfer of momentum, heat or moisture across the surface”. It can be a few tens of meters on a cool night and several kilometers on a hot afternoon; when it is low, pollution emitted or brought near the ground stays concentrated. Daily mean of hourly values. July 2024: 21.3 to 2,610 m
t2m	longitude, latitude, valid_time	K	“2 metre temperature”: the temperature of the air 2 m above land, sea, or inland water, in kelvin (subtract 273.15 for °C). Daily mean of hourly values. July 2024: 277.5 to 314.2 K
d2m	longitude, latitude, valid_time	K	“2 metre dewpoint temperature”: the temperature to which the air 2 m above the surface would have to cool to become saturated; a measure of humidity (the closer to t2m, the more humid). Daily mean of hourly values. July 2024: 258.1 to 301.4 K
u10	longitude, latitude, valid_time	m s**-1 (m/s)	“10 metre U wind component”: the eastward component of the wind 10 m above the surface; negative = toward the west. Daily mean of hourly values. July 2024: −11.7 to 10.2 m/s
v10	longitude, latitude, valid_time	m s**-1 (m/s)	“10 metre V wind component”: the northward component of the wind 10 m above the surface; negative = toward the south. Daily mean of hourly values. July 2024: −16.5 to 15.1 m/s
number	none	1	Ensemble member id, always 0, as in B1

Cleaning steps

You start from the clean county-day files, not from the raw grids. The research project’s script 02_winds.R turned each daily grid into county values, and the instructor exported them (see the appendix). For each county it took the 0.25° cells that contain the county’s 2020 block-group centers of population (the same block groups as in module m00) and averaged them, weighting each cell by the population whose block-group centers fall in it. Cells where nobody lives, such as lakes, therefore get no weight. It also converted the units: kelvin to °C, J/m² per hour to W/m², and geopotential to gpm.

The script R/m09_era5.R then makes these decisions:

1. Read one year per file. The 20 county-day files hold about 1.1 million rows each. You read only the years you need with map() and read_csv(), and the monthly table is built one year at a time, so the whole script needed about 1.7 GB of memory at its peak and about 1.5 minutes in our run. The chunk m09-month-build first removes the three years of daily rows, because the monthly loop does not need them.
2. Direction from u and v. Weather services report the direction the wind blows from, in degrees clockwise from north: 0° (or 360°) from the north, 90° from the east, 180° from the south, 270° from the west. The formula is (270 - atan2(v, u) * 180 / pi) %% 360. A calm day (u and v both 0.00 m/s after rounding) has no direction and gets NA.
3. Eight compass sectors. Each sector is 45° wide and centered on its direction: N covers 337.5–22.5°, NE 22.5–67.5°, and so on.
4. Northerly days. A day counts as northerly when the daily-mean 10 m wind comes from 315° through 45°, that is, from anywhere between northwest and northeast. This quarter of the compass covers the N sector and half of the NW and NE sectors. Calm days are not counted.
5. Average u and v, not directions. Monthly means of u10_ms and v10_ms give the average wind arrow. Averaging directions in degrees would be wrong: the mean of 350° and 10° is 180°, the opposite of north. The monthly ws10_ms is the mean of the daily speeds.
6. Same precision as the daily files. Monthly means are rounded like the daily values: 2 decimals for temperatures and winds, 1 for humidity, radiation, and heights, and whole meters for the boundary layer.

The first chunk works through u and v with five made-up winds. Each wind is a pair of numbers; the chunk turns each pair into a speed and a direction.

# ERA5 stores wind as two numbers, not as a speed and a direction.
#   u = the part of the wind blowing toward the east  (negative: toward the west)
#   v = the part of the wind blowing toward the north (negative: toward the south)
# Five made-up winds, in meters per second (m/s), show how the pair works.
wind_examples <- tribble(
  ~description,                        ~u10_ms, ~v10_ms,
  "west wind: blows toward the east",        5,       0,
  "north wind: blows toward the south",      0,      -5,
  "northwest wind: blows toward the SE",     4,      -4,
  "south wind: blows toward the north",      0,       3,
  "east wind: blows toward the west",       -2,       0
)

# Speed is the length of the arrow (Pythagoras). Direction follows the
# weather-service rule: the compass direction the wind comes FROM, in degrees
# clockwise from north (0 = from the north, 90 = east, 180 = south,
# 270 = west). atan2(v, u) is the angle of the arrow in radians, measured
# counterclockwise from east. "270 minus that angle in degrees" turns the
# direction the air moves TO into the direction it comes FROM, and %% 360
# wraps the result into 0-360.
wind_examples |>
  mutate(
    speed_ms = sqrt(u10_ms^2 + v10_ms^2),
    from_deg = (270 - atan2(v10_ms, u10_ms) * 180 / pi) %% 360
  )

The same formula, wrapped in two functions, is tested on the same five winds. The sectors should read W, N, NW, S, E.

# The same formula as a function, for the real data below. When u and v are
# both exactly 0 the air is calm and has no direction, so the function
# returns NA (atan2(0, 0) would otherwise give a meaningless 270).
wind_from_deg <- function(u, v) {
  if_else(u == 0 & v == 0, NA_real_, (270 - atan2(v, u) * 180 / pi) %% 360)
}

# Eight compass sectors, each 45 degrees wide and centered on its direction:
# "N" covers 337.5-22.5 degrees, "NE" 22.5-67.5, and so on. Adding 22.5 first
# shifts "N" to 0-45, so whole-number division by 45 (%/%) gives the sector
# number 0-7.
compass_sector <- function(deg) {
  sectors <- c("N", "NE", "E", "SE", "S", "SW", "W", "NW")
  factor(sectors[((deg + 22.5) %% 360) %/% 45 + 1], levels = sectors)
}

# Test both on the made-up winds: the sectors should read W, N, NW, S, E.
wind_examples |>
  mutate(
    from_deg = wind_from_deg(u10_ms, v10_ms),
    sector = compass_sector(from_deg)
  )

This function reads one year’s file. The column types keep the leading zero of GEOID and read date as a date.

# One clean file per year, each about 1.1 million rows (3,108 counties x 365
# or 366 days). Reading GEOID as text keeps its leading zero ("01001"); every
# weather column is a number.
read_era5_year <- function(yr) {
  read_csv(
    file.path(clean_dir, "era5_county_day", str_c("era5_county_day_", yr, ".csv.gz")),
    col_types = cols(GEOID = col_character(), date = col_date(),
                     .default = col_double())
  )
}

Read three years and check the grain: every county on every day, with no duplicates and no missing values.

# Pick a year range. Three years are about 3.4 million rows and 350 MB of
# memory. All 20 years at once would need more than 2 GB, so the monthly
# table further down is built one year at a time instead.
years <- 2021:2023
era5_days <- map(years, read_era5_year) |> list_rbind()

# Grain: one row per county x day, every county on every day, no missing values.
all_days <- seq(as.Date(str_c(min(years), "-01-01")),
                as.Date(str_c(max(years), "-12-31")), by = "day")
stopifnot(
  nrow(era5_days) == 3108 * length(all_days),
  nrow(distinct(era5_days, GEOID, date)) == nrow(era5_days),
  n_distinct(era5_days$GEOID) == 3108,
  setequal(era5_days$date, all_days),
  !anyNA(era5_days)
)
glimpse(era5_days)

Add the direction and sector to every county-day. Only 13 of the 3,403,260 county-days in 2021–2023 are calm.

# Add the direction the 10 m wind comes from, and its compass sector.
era5_days <- era5_days |>
  mutate(
    wind_from_deg = wind_from_deg(u10_ms, v10_ms),
    wind_sector = compass_sector(wind_from_deg)
  )

# Calm county-days (u and v both rounded to 0.00 m/s) have no direction.
# They are very rare; every other day gets a sector.
era5_days |>
  summarize(calm_days = sum(is.na(wind_from_deg)),
            min_deg = min(wind_from_deg, na.rm = TRUE),
            max_deg = max(wind_from_deg, na.rm = TRUE))

Look at one county during the June 2023 smoke. On June 2–8 the wind in Livingston County, NY, came from the north, northeast, or northwest every day.

# Livingston County, NY (GEOID 36051), home of SUNY Geneseo, in early June
# 2023, when smoke from fires in Quebec drifted over New York. Note the
# negative v (air moving south) on June 2-8, with winds from N, NE, or NW.
era5_days |>
  filter(GEOID == "36051",
         between(date, as.Date("2023-06-01"), as.Date("2023-06-12"))) |>
  select(date, t2m_c, rh_pct, u10_ms, v10_ms, ws10_ms, wind_from_deg,
         wind_sector, blh_m, ssrd_wm2)

A wind rose counts days by the direction the wind came from.

# A wind rose: on how many days the daily wind came from each sector in
# Livingston County, 2021-2023. Each bar points to where the wind came
# from; start = -pi / 8 turns the plot so that "N" sits at the top.
era5_days |>
  filter(GEOID == "36051", !is.na(wind_sector)) |>
  count(wind_sector) |>
  ggplot(aes(wind_sector, n)) +
  geom_col(fill = "steelblue", width = 0.9) +
  coord_polar(start = -pi / 8) +
  labs(x = NULL, y = "Days")

This function turns one year of county-days into county-months.

# Monthly means, one year at a time: read a year (1.1 million rows), shrink it
# to 37,296 county-months, and let R free the daily rows before the next year.
# u and v are averaged as plain numbers, which gives the average wind arrow.
# ws10_ms is the average of the daily speeds. It is never smaller than the
# length of the average arrow, because winds from opposite directions cancel
# in u and v but not in speed.
summarize_era5_year <- function(yr) {
  read_era5_year(yr) |>
    mutate(
      year = year(date),
      month = month(date),
      from_deg = wind_from_deg(u10_ms, v10_ms),
      # Northerly: the wind comes from anywhere between northwest (315) and
      # northeast (45), a quarter of the compass centered on north.
      northerly = from_deg >= 315 | from_deg <= 45
    ) |>
    summarize(
      n_days = n(),
      across(c(t2m_c, d2m_c, rh_pct, u10_ms, v10_ms, ws10_ms, blh_m, ssrd_wm2,
               z500_gpm, z500_anom_gpm), mean),
      # Calm days have no direction (NA), so they are not counted as northerly.
      days_north_wind = sum(northerly, na.rm = TRUE),
      .by = c(GEOID, year, month)
    )
}

Drop the daily table from memory, run the function for all 20 years, then round.

# The 3.4 million daily rows above are no longer needed. Removing them before
# the 20-year loop frees about 370 MB, so the loop has room for each year.
rm(era5_days)
invisible(gc())

# All 20 years, then rounding to the same precision as the daily files.
era5_months <- map(2006:2025, summarize_era5_year) |>
  list_rbind() |>
  mutate(
    across(c(t2m_c, d2m_c, u10_ms, v10_ms, ws10_ms), \(x) round(x, 2)),
    across(c(rh_pct, ssrd_wm2, z500_gpm, z500_anom_gpm), \(x) round(x, 1)),
    blh_m = round(blh_m)
  ) |>
  arrange(GEOID, year, month)

Check the grain and write the monthly file.

# Grain: one row per county x month, 3,108 counties x 240 months (2006-2025),
# complete, no missing values. Every month has all its days, northerly days
# fit inside the month, and the mean speed is at least the length of the
# mean arrow (0.02 m/s of slack for rounding).
stopifnot(
  nrow(era5_months) == 3108 * 240,
  nrow(distinct(era5_months, GEOID, year, month)) == nrow(era5_months),
  !anyNA(era5_months),
  all(era5_months$n_days == days_in_month(make_date(era5_months$year,
                                                    era5_months$month))),
  all(between(era5_months$days_north_wind, 0, era5_months$n_days)),
  all(era5_months$ws10_ms >=
        sqrt(era5_months$u10_ms^2 + era5_months$v10_ms^2) - 0.02)
)

write_csv(era5_months, file.path(clean_dir, "era5_county_month_2006_2025.csv"),
          na = "")

Finally, map the number of northerly days in June 2023. The county points come from counties_2020.csv (module m00).

# Days with a northerly surface wind in June 2023, one dot per county, placed
# at the county's population-weighted center (from module m00).
counties <- read_csv(file.path(clean_dir, "counties_2020.csv"),
                     col_types = cols_only(GEOID = col_character(),
                                           lat = col_double(), lon = col_double()))

era5_months |>
  filter(year == 2023, month == 6) |>
  left_join(counties, by = "GEOID") |>
  ggplot(aes(lon, lat, color = days_north_wind)) +
  geom_point(size = 0.6) +
  scale_color_viridis_c(name = "Northerly\ndays") +
  coord_quickmap() +
  labs(x = NULL, y = NULL)

Clean files

The daily files come from the instructor export (see the appendix); the monthly file is written by the code above.

File (in wildfire_data/clean/)	Rows	One row is	Size
era5_county_day/era5_county_day_<year>.csv.gz, 20 files for 2006–2025	1,134,420 per year (1,137,528 in 2008, 2012, 2016, 2020, 2024); 22,703,940 in all	a county × UTC day (instructor export)	28.5–28.7 MB each (about 80 MB uncompressed); 571.5 MB in all
era5_county_month_2006_2025.csv	745,920	a county × month, January 2006 to December 2025 (3,108 × 240)	53.0 MB

The monthly file is slightly above the 50 MB size at which this guide usually switches to .csv.gz. It stays an uncompressed .csv because the course plan names it that way, and 53 MB still opens quickly in R.

Clean variables

The 20 daily files share these columns. Rows are sorted by GEOID and date, and no cell is missing. Every weather value is a population-weighted county mean of the daily (UTC-day) means of hourly ERA5 values. read_csv() guesses these types correctly (the leading zeros keep GEOID as text), but read_era5_year() sets them explicitly so that nothing depends on the guess.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (2-digit state + 3-digit county), 2020 boundaries; joins to counties_2020.csv
date	date	—	UTC day of the daily mean (from 8 pm the evening before to 8 pm in New York in summer)
t2m_c	dbl	°C	Air temperature 2 m above the ground (raw t2m − 273.15). 2 decimals
d2m_c	dbl	°C	Dewpoint temperature 2 m above the ground (raw d2m − 273.15); never above t2m_c. 2 decimals
rh_pct	dbl	%	Relative humidity, 0–100. For each grid cell, 100 × e(d2m) / e(t2m), where e() is the saturation vapor pressure from the Magnus formula, capped at 100, then averaged over the county. It uses the daily-mean temperature and dewpoint, so it is not the same as the average of hourly humidity. 1 decimal
u10_ms	dbl	m/s	Eastward wind 10 m above the ground (raw u10); negative = toward the west. 2 decimals
v10_ms	dbl	m/s	Northward wind 10 m above the ground (raw v10); negative = toward the south. 2 decimals
ws10_ms	dbl	m/s	Wind speed of the daily-mean wind: √(u10_ms² + v10_ms²), computed from the unrounded county means. On a day when the wind turns, winds from opposite directions cancel, so this can be well below the average of the hourly speeds (it is never above it). 2 decimals
blh_m	dbl	m	Boundary-layer height: the depth of the well-mixed air next to the ground, averaged over the 24 hours (raw blh). Whole meters
ssrd_wm2	dbl	W/m²	Sunlight reaching the ground, averaged over all 24 hours, nights included (raw ssrd / 3,600). 1 decimal
z500_gpm	dbl	gpm	Height of the 500 hPa pressure level (raw z / 9.80665): 4,907 to 6,019 gpm in 2006–2025, lower in winter and farther north. 1 decimal
z500_anom_gpm	dbl	gpm	z500_gpm minus the county’s normal height for that day of the year. The normal is a smooth seasonal curve (a constant plus sine and cosine waves with periods of 1 year, 6 months, and 4 months) fitted to the county’s 1991–2005 days. No trend is removed, so later years tend to be positive (see the cautions). Positive = higher than normal (a ridge overhead). 1 decimal

era5_county_month_2006_2025.csv: one row per county × month, sorted by GEOID, year, and month, with no missing cells. read_csv() reads year, month, n_days, and days_north_wind as dbl.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code, 2020 boundaries
year	dbl	—	Calendar year, 2006 to 2025 (of the UTC day)
month	dbl	—	Calendar month, 1 (January) to 12 (December)
n_days	dbl	days	Days in the month (28–31); every day is present
t2m_c	dbl	°C	Monthly mean of daily t2m_c. 2 decimals
d2m_c	dbl	°C	Monthly mean of daily d2m_c. 2 decimals
rh_pct	dbl	%	Monthly mean of daily rh_pct. 1 decimal
u10_ms	dbl	m/s	Monthly mean of daily u10_ms (the eastward part of the average wind arrow). 2 decimals
v10_ms	dbl	m/s	Monthly mean of daily v10_ms (the northward part of the average wind arrow). 2 decimals
ws10_ms	dbl	m/s	Mean wind speed: monthly mean of daily ws10_ms. At least √(u10_ms² + v10_ms²), the speed of the average arrow. 2 decimals
blh_m	dbl	m	Monthly mean of daily blh_m. Whole meters
ssrd_wm2	dbl	W/m²	Monthly mean of daily ssrd_wm2. 1 decimal
z500_gpm	dbl	gpm	Monthly mean of daily z500_gpm. 1 decimal
z500_anom_gpm	dbl	gpm	Monthly mean of daily z500_anom_gpm. 1 decimal
days_north_wind	dbl	days	Days in the month whose daily-mean 10 m wind came from the north: from 315° through 45° (northwest through north to northeast). Calm days are not counted. 0 to n_days

Checks and cautions

• Daily files match the research project. R/check_m09.R compares all 22,703,940 county-days with the project’s Parquet tables (sl_county_day and z500_county_day). Every county is present on every day of 2006–2025, no GEOID–date pair is duplicated, no value is missing, and every value equals the project’s value rounded to the decimals listed above. d2m_c is never above t2m_c.
• Monthly file matches the research project. Monthly 500 hPa means differ from the project’s own county-month table (z500_county_month.parquet) by at most 0.074 gpm. Surface means differ from means recomputed from the project’s unrounded county-days (checked for 2006, 2012, 2023, and 2025) by at most 0.007 °C, 0.007 m/s, 0.07 percentage points of humidity, 0.07 W/m², and 0.7 m of boundary-layer height. These small gaps come from averaging rounded daily values and rounding again.
• Northerly days near the edges. A direction computed from rounded u and v can fall on the other side of 315° or 45° than the direction from unrounded values. Compared with directions from the project’s unrounded county-days, 98.1% of the 149,184 county-months checked have the same days_north_wind, and none differs by more than 2 days.
• Direction functions. On 100,000 random winds, wind_from_deg() agrees with an independent formula (the bearing the wind blows toward, atan2(u, v), plus 180°) to within 10⁻¹³ degrees, and compass_sector() always agrees with rounding the direction to the nearest multiple of 45°. A direction exactly on a sector border goes to the next sector clockwise: 22.5° is NE and 337.5° is N.
• Sector and “northerly” differ. The NW sector covers 292.5–337.5°, but only 315° and above counts as northerly. On June 6, 2023, the wind in Livingston County came from 301°: sector NW, not northerly. The county had 6 northerly days that June (June 2, 3, 4, 5, 7, and 8).
• Calm days. 98 county-days in 2006–2025 have u10_ms = v10_ms = 0.00 after rounding. Their direction and sector are NA, and they are not counted as northerly.
• ws10_ms is the speed of the average wind. In the daily files it differs from √(u10_ms² + v10_ms²) of the rounded columns by at most 0.012 m/s. Because the arrow is averaged before its length is taken, it is lower than the average hourly wind speed on days when the wind turns. For the same reason, the monthly ws10_ms (the mean of daily speeds) exceeds the speed of the monthly mean arrow by 1.55 m/s in the median county-month.
• Northerly winds are more common in the cold season. Over 2006–2025, 20.9% of county-days have a northerly daily wind: 25.0% of February days and 15.1% of July days. In June 2023, counties averaged 6.0 northerly days, and 6.5 in New York, Pennsylvania, New Jersey, and New England. Counties near the coast of northern California and Oregon had the most (up to 26 days, in Humboldt County, California): summer winds along that coast usually blow from the north.
• ERA5 agrees with station-based temperatures. Compared with nClimGrid daily mean temperature (module m04) on the 1,134,420 county-days of 2023, t2m_c has a correlation of 0.980, is 0.62 °C warmer on average, and differs by 1.62 °C on average (6.2 °C or less on 99% of county-days). Part of the gap is definitions: ERA5 averages 24 hourly values over a UTC day, while nClimGrid’s mean is (maximum + minimum) / 2 over the observers’ reporting day.
• UTC days, not local days. For U.S. counties the UTC day starts 4–8 hours before local midnight (8 pm EDT the evening before in New York, 5 pm PDT in California). Weather on a local evening therefore belongs to ERA5’s next date. Keep this in mind when you match ERA5 days with daily smoke or pollution data; monthly means are hardly affected.
• 500 hPa anomalies drift upward. The normal heights come from 1991–2005 and no trend is removed, so later years run high: the mean anomaly over 2006–2025 is +9.2 gpm, and yearly means range from −9.8 gpm (2008) to +22.2 gpm (2024). Compare anomalies within a period, or include year effects in a regression.
• Grid-cell values, not stations. The cells are about 18–28 km wide, so a county value smooths over valleys, lake shores, and cities and will differ from a nearby weather station. A coastal block-group center can sit in a cell that is partly sea; the research project notes that such cells blend land and sea values.
• No missing rows. Both files are complete: every county has a row for every day (daily files) and every month (monthly file), and no cell is empty. There is nothing to interpret as zero or unobserved.
• No county winds aloft. The 850 and 700 hPa winds (bundle B1) describe how smoke moves above the surface. The research project keeps them only as gridded files (data/cleaned/winds/winds_layer/winds_layer_<year>.nc: the mean of the two levels, shifted to each place’s local solar day, the day as the sun defines it at that longitude). The bundle has no county table of them.

Smoke episodes and Canadian fire shares

These four tables come from the course’s research project, not from an outside data provider. The project’s script 06_episode_count.R combines daily county smoke PM2.5 from Colorado State University (CSU), satellite fire detections from NASA’s MODIS instruments, and SEER county populations. It lists smoke episodes (two or more days in a row when a region’s population-weighted smoke PM2.5 is at least 5 µg/m³), counts person-days at each smoke level, and, for every smoky county-day, measures how much of the fire activity within 1,500 km was in Canada. Fire activity is measured by fire radiative power (FRP): the heat that burning vegetation gives off as radiation, seen by satellite and measured in megawatts (MW).

In this project the tables show when, where, and how often heavy smoke came with mostly Canadian fire activity: you can rank and date episodes, compare regions and years, and pick case studies such as June 2023. They cannot tell you where a county’s smoke actually came from. The Canadian share describes the fires that burned nearby, not the path the smoke took, so it is a descriptive attribution proxy, not a causal measure. The smoke values are model estimates, and the tables hold no health or other outcome data.

Provider	The course research project, “The cross-border cost of Canadian wildfire smoke” (SUNY Geneseo): script scripts/R/06_episode_count.R, build of 27 September 2026. Inputs: CSU kriged smoke PM2.5 (Dryad, version 8); NASA FIRMS MODIS active fire detections (Collection 6.0 through 2022, 6.1 from 2023); NCI SEER county populations
Coverage	2006–2025, daily · the 3,108 contiguous-U.S. counties (DC included), grouped into 9 regions; fires in Canada, the U.S. (including Alaska), and Mexico
Raw unit	none: this module ships no raw files (see “Raw files”)
Clean unit	one row = a region × smoke episode; a region × year; a region × year × smoke bin; a county × day with positive smoke PM2.5
Licence and citation	Derived tables; cite their inputs. Smoke: Pierce, J., Ford, B., O’Dell, K., McGinnis, J., & Fischer, E. (2026). Spatially interpolated non-smoke and smoke PM2.5 concentrations for the US from 2006-2025 (Version 8) [Dataset]. Dryad. https://doi.org/10.5061/dryad.k0p2ngfhv (CC0 1.0). Fires: NASA Fire Information for Resource Management System (FIRMS), MODIS active fire detections (MCD14ML), https://doi.org/10.5067/FIRMS/MODIS/MCD14ML (open, no restrictions). Population: National Cancer Institute, SEER Program, U.S. County Population Data, 1969–2024, https://seer.cancer.gov/popdata/. The CSU, FIRMS, and SEER sections give full citations
Documentation	Instructor-export appendix of this guide (every column of the project files); CSU dataset and README; NASA FIRMS; MODIS fire product; SEER population data; Census regions and divisions map

Raw files

This module ships no raw files. The instructor export reads the research project’s finished tables (see the appendix). The project built them from four inputs, and other sections of this guide work with each one from its raw data:

Input to the project build	What the project used	Where to see it in this guide
Smoke PM2.5	CSU kriged smoke, median background, population-weighted county means, 2006–2025	CSU section (raw 2023 file; all years in its instructor export)
Fire power	MODIS Terra and Aqua fire detections from one FIRMS archive delivery (2003–2025): vegetation fires with confidence of at least 30, summed by 1° cell and local solar day, with the project’s fill for days when a satellite was out	FIRMS section (yearly country files, 2003–2024, no outage fill)
Population	SEER July 1 county totals, 2006–2024; the 2024 counts stand in for 2025	SEER section
Regions and county points	Census regions, the project’s eastern split, and the Census internal point of each county	Places section (counties_2020.csv; its lat and lon are population centers, not internal points)

Raw variables

None: no raw files ship with this module. The appendix describes every column of the project files that the export reads.

Cleaning steps

The project script 06_episode_count.R built the tables, and the instructor export (R/m11x_instructor_export.R, see the appendix) renamed, rounded, and checked them. These decisions are behind every number:

1. Smoke means CSU smoke PM2.5 above zero. A county-day’s smoke is CSU smoke PM2.5 (median background), averaged over the county’s grid cells with population weights. Only positive values count as smoke; zero and negative values count as “no smoke”. Negative values occur because kriging, the statistical method CSU uses to fill in the map between air monitors, can undershoot. The CSU authors advise keeping negatives when you average, so regional means here are a little higher than means that keep them.
2. Nine regions of two types. “Census region”: the four Census Bureau regions, limited to the contiguous states and DC. Northeast: CT ME MA NH NJ NY PA RI VT. Midwest: IA IL IN KS MI MN MO ND NE OH SD WI. South: AL AR DC DE FL GA KY LA MD MS NC OK SC TN TX VA WV. West: AZ CA CO ID MT NM NV OR UT WA WY. Alaska and Hawaii are in no region, because the county universe leaves them out. “Eastern split”: four groups of eastern states chosen by the project, New England (CT ME MA NH RI VT), Mid-Atlantic (DE DC MD NJ NY PA), Great Lakes (IL IN MI OH WI), and Upper Midwest (IA MN ND SD), plus their union, “East (4 splits)”. The project’s Mid-Atlantic is not the Census Middle Atlantic division: it adds DE, DC, and MD, which the Census puts in the South. A county can belong to three regions at once, such as Northeast, Mid-Atlantic, and East (4 splits). States outside the four eastern groups have east_split = Other in counties_2020.csv and belong to no eastern-split region.
3. Population weights. Each county-day counts with the county’s SEER July 1 population of that year; the 2024 counts stand in for 2025. A region’s population is the sum over its counties. A person-day is one resident on one day.
4. Nearby fire power. For each county-day, the project adds up MODIS FRP in all 1° × 1° grid cells (about 111 km north to south) whose center lies within 1,500 km of the county’s internal point (a point inside the county published by the Census Bureau). It keeps three sums: fires in Canada, in the U.S. (including Alaska), and in Mexico. A fire counts on its local solar day: UTC (Coordinated Universal Time, the clock of satellite data) shifted by one hour for every 15° of longitude, so each day runs from about local midnight to midnight. The project matches that day to the smoke day with the same date. A lag version adds FRP over the day and the three days before (d−3 to d), because smoke can take days to travel.
5. Canadian share and “Canadian-dominated”. The share is Canadian FRP ÷ (Canadian + U.S. + Mexican FRP). A county-day is Canadian-dominated when its share is at least 0.5. If no fire burned within 1,500 km that day, the same-day share is undefined (missing), and the day is not Canadian-dominated.
6. Episodes. For each region and day, population-weighted smoke = Σ (county smoke × county population) ÷ region population; counties without smoke add 0. A day is smoky when this is at least 5 µg/m³, and an episode is a run of two or more smoky days in a row. The episode’s Canadian share is a two-step average: each day, the population-weighted mean share over the region’s smoky counties that have a share; then the mean over the episode’s days, each day weighted by its population-weighted smoke. The episode is Canadian-dominated when this average is at least 0.5.
7. Person-days by smoke bin. The population of each smoky county-day goes into one bin: (0,5], (5,15], (15,35], (35,55], or above 55 µg/m³, where (5,15] means above 5 and at most 15. The “0 / no smoke” bin is everyone else: region population × days in the year, minus the smoky person-days. The 15 and 35 µg/m³ cut-offs echo the WHO 24-hour guideline and the U.S. EPA 24-hour standard for total PM2.5; here they apply to the smoke part alone.
8. What the export changed. Clearer, consistent column names (for example pd_gt15 became person_days_gt15); rounding (PM2.5 to 2 decimals, FRP to 1, shares to 3 or 4); the constant conc_source column dropped after a check that it names CSU; bin rows that the project left out (no person-days in that bin) added with zeros; and, in the fire-share file, the county’s smoke PM2.5 and two Canadian-dominated flags computed before rounding.

The first chunk reads the county file from the places section and prints the states in each region. Note that the eastern Mid-Atlantic group spans two Census regions, and that the regions overlap, so counts must not be added across them.

# The episode and summary files use nine regions of two types. "Census
# region" rows are the four Census regions. "Eastern split" rows are four
# groups of eastern states chosen by the research project, plus their union,
# "East (4 splits)". The two types overlap: a New York county counts in both
# "Northeast" and "Mid-Atlantic". All nine cover only the contiguous U.S. and
# DC (the West has no Alaska or Hawaii). The county file lists both
# groupings, so we can print the states in each group.
counties <- read_csv(
  file.path(clean_dir, "counties_2020.csv"),
  col_types = cols_only(
    GEOID = col_character(), state_abbr = col_character(),
    census_region = col_character(), east_split = col_character(),
    pop_2020 = col_double()
  )
)
stopifnot(nrow(counties) == 3108, !anyDuplicated(counties$GEOID))

counties |>
  distinct(census_region, east_split, state_abbr) |>
  summarize(states = str_c(sort(state_abbr), collapse = " "),
            .by = c(census_region, east_split)) |>
  arrange(census_region, east_split)

Read the episode list with explicit column types, then count episodes by region. Of the 512 episodes, 123 were Canadian-dominated; the South had none.

# One row per smoke episode: 2 or more days in a row on which a region's
# population-weighted smoke PM2.5 was at least 5 ug/m3. Declaring the column
# types makes the dates real Dates and the flag a TRUE/FALSE column.
episodes <- read_csv(
  file.path(clean_dir, "smoke_episodes_2006_2025.csv"),
  col_types = cols(
    episode_id = col_integer(), region_type = col_character(),
    region = col_character(), start_date = col_date(), end_date = col_date(),
    year = col_integer(), n_days = col_integer(),
    peak_smoke_pm25 = col_double(), share_ca_mean = col_double(),
    person_days_gt15 = col_double(), canadian_dominated = col_logical()
  )
)
stopifnot(nrow(episodes) == 512, !anyDuplicated(episodes$episode_id))

# How many episodes did each region have, and how many were
# Canadian-dominated (Canada supplied at least half the nearby fire power)?
# Regions overlap, and one big smoke event is an episode in every region it
# covers, so do not add these counts across regions.
episodes |>
  summarize(episodes = n(), canadian_dominated = sum(canadian_dominated),
            .by = c(region_type, region))

Example (a) lists the largest Canadian-dominated episodes in one region. In the eastern states, 7 of the 10 largest fell in 2023. The largest ran from 30 May to 11 June 2023: 13 days, a peak of 63.7 µg/m³, a Canadian share of 0.879, and 491 million person-days above 15 µg/m³.

# (a) The largest Canadian-dominated episodes in one region. "Largest" means
# the most person-days above 15 ug/m3: on each day, add up the residents of
# counties whose smoke PM2.5 was above 15, then add over the episode's days.
# Change my_region to any of the nine region names to see another region.
my_region <- "East (4 splits)"
stopifnot(my_region %in% episodes$region)

episodes |>
  filter(region == my_region, canadian_dominated) |>
  arrange(desc(person_days_gt15)) |>
  mutate(person_days_gt15_millions = round(person_days_gt15 / 1e6, 1)) |>
  select(start_date, end_date, n_days, peak_smoke_pm25, share_ca_mean,
         person_days_gt15_millions) |>
  slice_head(n = 10) |>
  print(width = Inf)

Counting Canadian-dominated episodes per year shows how unusual the recent years are: 6 to 8 per eastern group in 2023, against 0 to 2 per year before 2021.

# Canadian-dominated episodes per year in the four eastern groups. Years
# with no such episode get an explicit 0 from complete(), so the table shows
# every year from 2006 to 2025.
episodes |>
  filter(region_type == "Eastern split", region != "East (4 splits)",
         canadian_dominated) |>
  count(year, region) |>
  complete(year = 2006:2025, region, fill = list(n = 0)) |>
  pivot_wider(names_from = region, values_from = n)

The region-year summary turns episodes into totals for the year. In 2023 the average eastern resident had 10.7 Canadian-dominated days above 15 µg/m³, and 99.7% of eastern residents lived in a county with at least one such day. 2025 ranks second.

# One row per region x year: person-days above 15 and 35 ug/m3, and the part
# of them that fell on Canadian-dominated county-days.
ca_summary <- read_csv(
  file.path(clean_dir, "canadian_dominated_summary.csv"),
  col_types = cols(region_type = col_character(), region = col_character(),
                   year = col_integer(), .default = col_double())
)
stopifnot(nrow(ca_summary) == 9 * 20)

# The worst eastern years. days_gt15_ca_per_capita is the number of
# Canadian-dominated days above 15 ug/m3 for the average eastern resident.
ca_summary |>
  filter(region == "East (4 splits)") |>
  arrange(desc(person_days_gt15_ca)) |>
  select(year, person_days_gt15, person_days_gt15_ca, share_gt15_ca,
         days_gt15_ca_per_capita, pop_share_any_gt15_ca) |>
  slice_head(n = 5) |>
  print(width = Inf)

The bin file spreads each year over smoke levels. In 2023 the average eastern resident spent 220 days with no smoke and 2.4 days above 55 µg/m³.

# One row per region x year x smoke bin. days_per_capita is the number of
# days the average resident spent in each bin; the six bins of a
# region-year add up to 365 (366 in a leap year).
bins <- read_csv(
  file.path(clean_dir, "smoke_person_days_by_bin.csv"),
  col_types = cols(region_type = col_character(), region = col_character(),
                   year = col_integer(), smoke_bin = col_character(),
                   .default = col_double())
) |>
  # Keep the bins in order from no smoke to heaviest smoke.
  mutate(smoke_bin = fct_inorder(smoke_bin))
stopifnot(nrow(bins) == 9 * 20 * 6)

bins |>
  filter(region_type == "Eastern split", year == 2023) |>
  select(region, smoke_bin, days_per_capita) |>
  pivot_wider(names_from = smoke_bin, values_from = days_per_capita)

The county-day fire-share file is the module’s big file: 2.9 million rows, 58 MB compressed. read_csv() reads the .gz file directly. cols_only() both sets the column types (so GEOID stays text and the flag becomes TRUE/FALSE) and keeps only the 5 of 13 columns that the examples use, which saves memory.

# One row per county-day with positive smoke PM2.5, 2006-2025 (2.9 million
# rows). A county-day with no row had no smoke plume over its residents (CSU
# smoke exactly 0) or a slightly negative CSU estimate, so the project
# computed no fire share for it. A missing row says nothing about fires.
# The examples need 5 of the 13 columns. cols_only() reads just those, which
# saves memory; add any other column you need (the guide's "Clean variables"
# table lists all 13).
fire_share <- read_csv(
  file.path(clean_dir, "county_day_fire_share_2006_2025.csv.gz"),
  col_types = cols_only(
    GEOID = col_character(), date = col_date(), smoke_pm25 = col_double(),
    share_ca_sameday = col_double(), ca_dominated_sameday = col_logical()
  )
)
# Grain check: one row per county and date. Counting distinct key pairs
# needs far less memory than anyDuplicated() on a 2.9-million-row table.
stopifnot(nrow(fire_share) == 2876135,
          nrow(distinct(fire_share, GEOID, date)) == nrow(fire_share))

The fire-share file has a row only for smoky county-days. In June 2023 that is 73,887 of the 93,240 county-days (79.2%), and every one of them had some fire within 1,500 km.

# June 2023: how many of the 3,108 x 30 county-days had smoke, and so a row?
june_2023 <- fire_share |>
  filter(date >= ymd("2023-06-01"), date <= ymd("2023-06-30"))

tibble(
  rows = nrow(june_2023),
  possible = nrow(counties) * 30,
  pct_with_smoke = round(100 * rows / possible, 1),
  no_fire_nearby = sum(is.na(june_2023$share_ca_sameday))
)

Example (b) summarizes the Canadian share of nearby fire power by state for June 2023. It is a descriptive attribution proxy, not a causal measure: it says how Canadian the fire activity around a state’s smoky counties was, not how much of the smoke came from Canada. The chunk first joins each county-day to its state and stops if any county-day finds no county. It then computes two averages of share_ca_sameday. share_ca_popwt weights each smoky county-day by the county’s population, so a day with 0.1 µg/m³ of smoke counts as much as a day with 100 µg/m³. share_ca_smokewt weights each county-day by population × smoke PM2.5, so heavy-smoke days count more, as in the episode share share_ca_mean. The two differ when a state’s heaviest smoke came on days with less Canadian fire nearby: Illinois has 0.755 by population but 0.620 by smoke. Example (b) weights by the 2020 Census population in counties_2020.csv, not by the SEER population of each year that the episode and summary tables use; the difference is small for a single month. The two denominators that include days without a row (percent of county-days with smoke, and mean smoke over all 30 days) treat those days as zero smoke, which is what a missing row means. Minnesota (0.925), the New England states (0.91 to 0.96), and the Great Lakes states (0.73 to 0.89) had mostly Canadian fire activity nearby. In Texas, Florida, Louisiana, Oklahoma, Arizona, California, and New Mexico the share was below 0.001.

# (b) A DESCRIPTIVE ATTRIBUTION PROXY, NOT A CAUSAL MEASURE.
# share_ca_sameday is the part of all fire radiative power (FRP) burning
# within 1,500 km of a county that day that was in Canada. It does not trace
# the county's smoke back to a fire: winds can bring smoke from farther away
# or carry nearby smoke elsewhere. Read it as "how Canadian was the fire
# activity around this county on its smoky days".
# Population here is the 2020 Census count in counties_2020.csv, not the
# SEER population of each year that the episode and summary files use. For
# one month the difference is small.
june_2023_counties <- june_2023 |>
  left_join(counties, by = "GEOID", relationship = "many-to-one")
# Every county-day must find its county, or its state would be NA.
stopifnot(!anyNA(june_2023_counties$state_abbr))

state_share <- june_2023_counties |>
  summarize(
    smoke_county_days = n(),
    # Mean share over smoky county-days, each weighted by its county's
    # population. Days with no fire nearby (share missing) are dropped. A
    # day with 0.1 ug/m3 of smoke counts as much as a day with 100.
    share_ca_popwt = weighted.mean(share_ca_sameday, pop_2020, na.rm = TRUE),
    # The same mean with each county-day also weighted by its smoke, so
    # heavy-smoke days count more, as in the episode share.
    share_ca_smokewt = weighted.mean(share_ca_sameday, pop_2020 * smoke_pm25,
                                     na.rm = TRUE),
    # Percent of smoky person-days that were Canadian-dominated.
    pct_ca_dominated = 100 * sum(pop_2020 * ca_dominated_sameday) / sum(pop_2020),
    smoke_person_days = sum(smoke_pm25 * pop_2020),
    .by = c(state_abbr, census_region)
  )

# Add each state's county count and population to get two denominators that
# include the days without a row. Those days had no plume (smoke exactly 0)
# or a slightly negative estimate; like the project, we count them as 0.
state_share <- counties |>
  summarize(n_counties = n(), state_pop = sum(pop_2020),
            .by = c(state_abbr, census_region)) |>
  left_join(state_share, by = c("state_abbr", "census_region")) |>
  mutate(
    smoke_county_days = replace_na(smoke_county_days, 0),
    pct_county_days_smoky = round(100 * smoke_county_days / (n_counties * 30), 1),
    # Population-weighted mean smoke PM2.5 over all 30 days, zeros included.
    mean_smoke_pm25 = round(replace_na(smoke_person_days, 0) / (state_pop * 30), 2),
    share_ca_popwt = round(share_ca_popwt, 3),
    share_ca_smokewt = round(share_ca_smokewt, 3),
    pct_ca_dominated = round(pct_ca_dominated, 1)
  ) |>
  select(state_abbr, census_region, pct_county_days_smoky, mean_smoke_pm25,
         share_ca_popwt, share_ca_smokewt, pct_ca_dominated) |>
  arrange(desc(share_ca_popwt))

stopifnot(nrow(state_share) == 49, !anyDuplicated(state_share$state_abbr))
print(state_share, n = Inf, width = Inf)

The plot puts the population-weighted share next to the mean smoke. Run this chunk in RStudio (or render the guide) to see the plot: when the script runs with Rscript, the last line skips drawing so that R does not write a stray Rplots.pdf file. Use ggsave() to save a copy. The states with the most smoke in June 2023 (Wisconsin 24.0 µg/m³, Michigan 21.4, Pennsylvania 21.2) also had mostly Canadian fires nearby, while the southern states had smoke with almost no Canadian fire activity within 1,500 km.

# Each label is a state: how Canadian its nearby fire activity was on smoky
# days (x) against how much smoke its residents breathed in June 2023 (y).
# The pattern is descriptive: it shows where Canadian fires dominated the
# fire activity around smoky counties, not how much smoke they caused.
june_plot <- ggplot(state_share, aes(x = share_ca_popwt, y = mean_smoke_pm25,
                                     label = state_abbr, color = census_region)) +
  geom_text(size = 3) +
  labs(
    x = "Canadian share of same-day FRP within 1,500 km (population-weighted)",
    y = "Mean smoke PM2.5, June 2023 (µg/m³)",
    color = "Census region"
  ) +
  theme_minimal()

# Show the plot in RStudio or in a Quarto render. Run with Rscript, a plot
# would go to a stray file named Rplots.pdf, so the script draws it only when
# someone is watching. Use ggsave() if you want to keep a copy.
if (interactive() || isTRUE(getOption("knitr.in.progress"))) print(june_plot)

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
smoke_episodes_2006_2025.csv	512	a region × smoke episode	0.04 MB
canadian_dominated_summary.csv	180	a region × year (9 regions × 20 years)	0.02 MB
smoke_person_days_by_bin.csv	1,080	a region × year × smoke bin (9 × 20 × 6)	0.08 MB
county_day_fire_share_2006_2025.csv.gz	2,876,135	a county × day with positive smoke PM2.5 (12.7% of all 22.7 million county-days)	57.7 MB (214.6 MB unzipped)

Clean variables

smoke_episodes_2006_2025.csv: one row per region × episode, sorted by region_type, region, and start_date; no cell is missing. read_csv() reads episode_id, year, and n_days as dbl; the student code declares them as int.

Variable	Type	Units	Description
episode_id	dbl	—	Row number, 1 to 512, in the sort order above. Not stable if the project rebuilds the list
region_type	chr	—	Census region or Eastern split
region	chr	—	Midwest, Northeast, South, West (Census regions); New England, Mid-Atlantic, Great Lakes, Upper Midwest, or their union East (4 splits) (eastern split). Cleaning step 2 lists the states in each region; Alaska and Hawaii are in none
start_date	date	—	First smoky day of the episode
end_date	date	—	Last smoky day of the episode
year	dbl	—	Year of start_date. No episode crosses into a new year
n_days	dbl	days	Length: end_date − start_date + 1, at least 2. Every day in between was smoky (population-weighted smoke ≥ 5 µg/m³)
peak_smoke_pm25	dbl	µg/m³	Highest daily population-weighted smoke PM2.5 of the region during the episode; at least 5 by construction. 2 decimals
share_ca_mean	dbl	share, 0 to 1	Canadian share of same-day FRP within 1,500 km: population-weighted over the region’s smoky counties each day, then averaged over the episode’s days with each day weighted by its population-weighted smoke (cleaning step 6). 3 decimals
person_days_gt15	dbl	person-days	Sum over the episode’s days of the population of the region’s counties with smoke PM2.5 above 15 µg/m³ that day. Can be 0 (98 episodes)
canadian_dominated	lgl	—	TRUE when share_ca_mean ≥ 0.5 (decided before rounding; the rounded value gives the same answer in every row)

canadian_dominated_summary.csv: one row per region × year, complete (9 regions × 2006–2025). All columns except region_type and region are read as dbl.

Variable	Type	Units	Description
region_type	chr	—	Census region or Eastern split, as in the episode file
region	chr	—	One of the nine regions, as in the episode file
year	dbl	—	Calendar year
pop	dbl	persons	Region population: sum of the SEER July 1 populations of its counties (2024 counts for 2025)
person_days_gt15	dbl	person-days	Person-days with county smoke PM2.5 above 15 µg/m³, over every day of the year
person_days_gt15_ca	dbl	person-days	The part of person_days_gt15 on Canadian-dominated county-days (same-day share ≥ 0.5)
person_days_gt15_ca_lag03	dbl	person-days	The same with the lag share (FRP summed over days d−3 to d). It can be larger or smaller than the same-day value
person_days_gt35	dbl	person-days	Person-days with county smoke PM2.5 above 35 µg/m³
person_days_gt35_ca	dbl	person-days	The part of person_days_gt35 on Canadian-dominated county-days (same day)
days_gt15_ca_per_capita	dbl	days per resident	person_days_gt15_ca ÷ pop: Canadian-dominated days above 15 µg/m³ for the average resident. 3 decimals
share_gt15_ca	dbl	share, 0 to 1	person_days_gt15_ca ÷ person_days_gt15. Empty when the region had no person-days above 15 that year (11 region-years, all in the Northeast and its eastern groups). 3 decimals
pop_share_any_gt15_ca	dbl	share, 0 to 1	Share of the region’s population living in a county with at least one Canadian-dominated day above 15 µg/m³ that year. 3 decimals

smoke_person_days_by_bin.csv: one row per region × year × bin, complete; within each region-year the bins run from 0 / no smoke to >55. region_type, region, year, and pop are as in the summary file.

Variable	Type	Units	Description
region_type	chr	—	Census region or Eastern split
region	chr	—	One of the nine regions
year	dbl	—	Calendar year
smoke_bin	chr	µg/m³	Daily county smoke PM2.5 bin: 0 / no smoke (zero, negative, or no estimate above zero), (0,5], (5,15], (15,35], (35,55], >55. (a,b] means above a and at most b
person_days	dbl	person-days	Population × days in the bin. The six bins of a region-year add up to pop × 365 (366 in leap years). 0 in the 187 rows the export added
person_days_ca	dbl	person-days	The part of person_days on Canadian-dominated county-days (same day). Empty in the 0 / no smoke bin, where it does not apply
person_days_ca_lag03	dbl	person-days	The same with the lag share. Empty in the 0 / no smoke bin
person_days_no_fire	dbl	person-days	The part of person_days on county-days with no fire within 1,500 km that day (share undefined). Empty in the 0 / no smoke bin
pop	dbl	persons	Region population, as in the summary file
days_per_capita	dbl	days per resident	person_days ÷ pop: days the average resident spent in the bin. 3 decimals; a region-year’s six values add up to 365 or 366 within 0.002

county_day_fire_share_2006_2025.csv.gz: one row per county-day with positive smoke PM2.5, sorted by GEOID and date. read_csv() guesses every type correctly.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (state + county), 2020 boundaries; joins to counties_2020.csv
date	date	—	Day of the CSU smoke value (EPA daily averages cover the local calendar day). Fires are matched on their local solar day with the same date
smoke_pm25	dbl	µg/m³	CSU smoke PM2.5 of the county-day (population-weighted, median background), the same value as in the CSU files. Above 0 before rounding; 8,354 rows show 0.00 after rounding to 2 decimals
frp_ca_sameday	dbl	MW	Total FRP of Canadian fires in 1° cells whose center is within 1,500 km of the county’s internal point, on the same local solar day. 1 decimal
frp_us_sameday	dbl	MW	The same for U.S. fires (contiguous states and Alaska; Hawaii and the territories are outside the fire data)
frp_mx_sameday	dbl	MW	The same for Mexican fires
frp_ca_lag03	dbl	MW	Canadian FRP within 1,500 km, summed over the day and the 3 days before (d−3 to d). 1 decimal
frp_us_lag03	dbl	MW	The same for U.S. fires
frp_mx_lag03	dbl	MW	The same for Mexican fires
share_ca_sameday	dbl	share, 0 to 1	frp_ca_sameday ÷ (frp_ca_sameday + frp_us_sameday + frp_mx_sameday), from unrounded FRP. Empty when all three are 0, i.e., no fire within 1,500 km that day (398 rows). 4 decimals
share_ca_lag03	dbl	share, 0 to 1	The same from the three lag columns. Never empty. 4 decimals
ca_dominated_sameday	lgl	—	TRUE when the unrounded same-day share is at least 0.5; FALSE when it is lower or undefined. Use it instead of share_ca_sameday >= 0.5: rounding to 4 decimals moves 27 rows across 0.5
ca_dominated_lag03	lgl	—	The same for the lag share (29 rows would move with the rounded share)

Checks and cautions

• No outcome data. The validation script R/check_m11.R read the column names of every file in the project’s episodes folder (7 files, 101 columns, including the .rds summary object) and of the four exports. None holds mortality, hospital, or any other health outcome.
• Same as the project files. All 512 episodes, 180 region-years, and 893 bin rows of the project files are in the exports, with identical person-days and flags; PM2.5 differs by at most 0.005 µg/m³ and shares by at most 0.0005 (rounding). The fire-share file has the project’s 2,876,135 county-days key for key, and they are exactly the county-days with CSU smoke above 0. FRP differs by at most 0.05 MW, shares by at most 0.00005, and smoke_pm25 from the CSU panel by at most 0.005 µg/m³.
• Rebuilt from the definitions. Using the project’s CSU, SEER, and fire-share inputs and the steps above, the check rebuilds all 512 episodes with the same start, end, length, and person-days above 15, all 713 smoky bin rows, and all 180 values of pop_share_any_gt15_ca exactly. The bundle’s counties_2020.csv assigns every county to the same regions as the project, and the bundle’s SEER file gives the same region populations in all 180 region-years.
• You can rebuild the episodes yourself. From bundle files only (this module’s smoke_pm25 and share_ca_sameday, the SEER file, and counties_2020.csv), the same steps find the same 512 episodes with the same dates and flags. Peaks differ by at most 0.0014 µg/m³ and shares by at most 0.00004; person-days above 15 differ by up to 1.5 million in a few episodes, because rounding moves some county-days across 15.00.
• A missing row is not a zero-fire day. The fire-share file covers 12.7% of all county-days. A county-day without a row had smoke PM2.5 of zero or below, so the project computed no fire share for it (not applicable). A row with an empty share_ca_sameday is different: smoke was present but no fire burned within 1,500 km that day (398 rows).
• Canadian-dominated county-days. 313,614 county-days are Canadian-dominated by the same-day share and 361,452 by the lag share. The same-day flag can switch abruptly when a new fire flares; the project notes Georgia fires on 23–24 May 2007 and a Quebec flare on 25 May 2007. Compare both versions before you rely on one.
• What the share leaves out. The 1,500 km circle is a fixed rule, not a smoke path: it ignores winds, the height of the smoke, and fires farther away. FRP measures how hot fires burn at the moment a satellite passes, not how much smoke they release, and clouds or thick smoke can hide fires. Mexican fires count as non-Canadian. The project does not test the 1,500 km choice in these tables.
• Region means and empty regions. Episodes need a region-wide population-weighted mean of at least 5 µg/m³, so local smoke in a few counties does not make an episode. The South has 25 episodes and none is Canadian-dominated. The Northeast and its eastern groups had no person-days above 15 µg/m³ in some years (2014, 2016, 2017; New England also 2020 and 2022), so share_gt15_ca is empty there.
• Fire data differ from the FIRMS section. The project used one archive delivery of MODIS detections (2003–2025), with its own fill for days when one satellite was out (its column frp_interp_v2, Terra plus Aqua). The FIRMS section of this guide uses the yearly country files (2003–2024) without that fill, and the places file gives population centers, not the internal points used here. A rebuild of the shares from bundle files would come close but not match exactly.
• Smoke choices. The project uses the median background and treats negative CSU smoke as no smoke; the CSU README recommends the mean background for most uses and keeping negative values. The CSU releases differ by year (2006–2023: 2025-03-21; 2024: 2026-02-25; 2025: 2026-08-06).
• Population for 2025. SEER stops at 2024, so 2025 uses the 2024 county populations.
• Memory and time. R/m11_episodes.R runs in about 5 seconds on the instructor’s laptop and needs about 1.3 GB of memory, almost all of it for the fire-share file, even with only 5 columns read. Each added column costs more memory; if your computer is short on memory, filter to the years you need right after reading and remove fire_share with rm(fire_share) when you are done.

Putting it together: county panels

This section joins the clean files of the earlier sections into two analysis-ready tables. The monthly panel has one row for every county and month of 2006–2025, with county facts, population, four smoke measures (CSU, HMS plumes, Childs, EPA monitors), two weather sources (nClimGrid and ERA5), and Canada-wide fire activity. The daily panel has one row for every county and day of 2023, the year of the large Canadian smoke episodes, with daily smoke, monitor, model, weather, wind, and fire-share columns. No new data enter here: every number is copied from a file built in an earlier section, and each column name starts with a prefix that tells you which one.

Use the monthly panel for regressions and maps across years, and the daily panel for case studies such as the June 2023 smoke in the East. The panels cannot settle where a county’s smoke came from: CSU, HMS, Childs, and the monitors count smoke from all fires, and the Canadian measures (EMBER’s Canadian-fire PM2.5 and the nearby-fire share) are a model run and a distance rule. The panels also hold no health or other outcome data.

Provider	Built in this guide (R/m12_panel.R) from the clean files of the places, FIRMS, HMS, AQS, nClimGrid, Childs, SEER, CSU, EMBER, ERA5, and episodes sections. Each of those sections names its original provider
Coverage	Monthly panel: January 2006 – December 2025 · daily panel: 1 January – 31 December 2023 · both: the 3,108 counties of the contiguous U.S. and DC (2020 boundaries)
Raw unit	none: this module ships no raw files (see “Raw files”)
Clean unit	one row = a county × month (monthly panel); a county × day (daily panel)
Licence and citation	Derived tables; cite the sources of the columns you use. The licence and citation rows of each earlier section give them (for example, CSU smoke: Pierce et al. 2026, Dryad, CC0 1.0)
Documentation	The “Clean variables” and “Checks and cautions” parts of each earlier section of this guide, which describe every input column; dplyr left_join() and its relationship argument; readr read_csv_chunked()

Raw files

This module ships no raw files. It reads only clean files written by the earlier sections, all in wildfire_data/clean/:

Clean input	Section	Used in	How it joins
counties_2020.csv	Places	both panels	builds the frame: every GEOID
county_cbsa_2023.csv	Places	monthly	GEOID, many-to-one
seer_county_year_2006_2024.csv	SEER	monthly	GEOID + year, many-to-one
csu_smoke_county_month_2006_2025.csv	CSU	monthly	GEOID + year + month, one-to-one
hms_county_month_2006_2025.csv	HMS	monthly	GEOID + year + month, one-to-one
childs_smoke_county_month_2006_2023.csv	Childs	monthly	GEOID + year + month, one-to-one
aqs_county_month_2006_2025.csv	AQS	monthly	GEOID + year + month, one-to-one
nclimgrid_county_month_2006_2025.csv	nClimGrid	monthly	GEOID + year + month, one-to-one
era5_county_month_2006_2025.csv	ERA5	monthly	GEOID + year + month, one-to-one
firms_canada_province_day_2003_2024.csv	FIRMS	monthly	summed to Canada × month, then year + month, many-to-one
csu_smoke_county_day/csu_smoke_county_day_2023.csv.gz	CSU	daily	GEOID + date, one-to-one
hms_calendar_2005_2025.csv	HMS	daily	date, many-to-one
hms_county_day_2006_2025.csv.gz	HMS	daily	GEOID + date, one-to-one (2023 rows only)
aqs_county_day_2006_2025.csv.gz	AQS	daily	GEOID + date, one-to-one (2023 rows only)
ember_county_day_2023.csv	EMBER	daily	GEOID + date, one-to-one
nclimgrid_county_day/nclimgrid_county_day_2023.csv.gz	nClimGrid	daily	GEOID + date, one-to-one
era5_county_day/era5_county_day_2023.csv.gz	ERA5	daily	GEOID + date, one-to-one
childs_smoke_county_day/childs_smoke_county_day_2023.csv.gz	Childs	daily	GEOID + date, one-to-one
county_day_fire_share_2006_2025.csv.gz	Episodes	daily	GEOID + date, one-to-one (2023 rows only)

Raw variables

None: no raw files ship with this module. The earlier sections describe every column of the clean inputs.

Cleaning steps

The script makes these decisions:

1. Start from a complete frame. Each panel begins as every county crossed with every month (3,108 × 240 = 745,920 rows) or every day of 2023 (3,108 × 365 = 1,134,420 rows). Every source is attached with left_join(), which keeps all rows of the frame. A county-month or county-day that a source does not cover keeps its row and gets NA (missing) in that source’s columns, so no county silently drops out.
2. Declare each join’s relationship. relationship = "one-to-one" means each frame row matches at most one source row, and the reverse. "many-to-one" means many frame rows share one source row: one county-year of population fills 12 county-months, and one Canada-wide month fills all 3,108 counties. If a key were repeated where it should not be, left_join() would stop with an error instead of duplicating rows. After every join, stopifnot() checks that the row count has not changed.
3. Prefix every column by its source. pop_ SEER, csu_ CSU, hms_ HMS, childs_ Childs, aqs_ EPA AQS monitors, wx_ nClimGrid weather, era5_ ERA5, ember_ EMBER, fire_ca_ FIRMS (all of Canada), and nearfire_ the nearby-fire Canadian share of the episodes section. County facts (county_name, state_abbr, …) and n_days have no prefix. A few columns are renamed to fit (for example CSU’s total_pm25_mean becomes csu_total_pm25); the variable tables give the original name.
4. Fill a missing row only when the answer is known. EPA day counts become 0 where a county had no monitor reading (0 days measured is true), but monitor means stay NA. HMS plume columns become 0 on analysed days without a plume, and stay NA on the one 2023 day nobody analysed. Everything else that is missing stays NA, with one meaning per column (see the variable tables).
5. Hold 2025 population flat. SEER ends in 2024, so 2025 rows repeat the 2024 numbers, and pop_carried_forward is TRUE on them.
6. Drop duplicate bookkeeping. Each monthly source carries its own day count. The script checks that HMS, nClimGrid, and ERA5 agree with the frame’s n_days and keeps only that one. CSU’s n_days_obs, complete_month, and csu_release are dropped after checking that every month is complete (the CSU section lists the release of each year).
7. Read only 2023 from the long daily files. The HMS, AQS, and fire-share county-day files cover 2006–2025. read_csv_chunked() reads them 500,000 rows at a time and keeps only the 2023 rows, so the whole file never sits in memory.

The first chunk builds the monthly frame. The CBSA type (metropolitan or micropolitan area) comes along here; a county in no CBSA gets "none", because that is a real answer, not missing data.

# Every panel starts from a complete frame: each of the 3,108 counties in
# every month of 2006-2025. Joining the sources onto this frame (instead of
# joining sources to each other) means a county-month that a source leaves
# out still has a row, with NA in that source's columns.
counties <- read_csv(
  file.path(clean_dir, "counties_2020.csv"),
  col_types = cols_only(
    GEOID = col_character(), county_name = col_character(),
    state_abbr = col_character(), census_region = col_character(),
    census_division = col_character(), east_split = col_character()
  )
)
# A county outside any metro or micro area has an empty cbsa_type. That is a
# real answer, not missing data, so we spell it out as "none".
cbsa <- read_csv(
  file.path(clean_dir, "county_cbsa_2023.csv"),
  col_types = cols_only(GEOID = col_character(), cbsa_type = col_character())
) |>
  mutate(cbsa_type = replace_na(cbsa_type, "none"))
stopifnot(nrow(counties) == 3108, !anyDuplicated(counties$GEOID),
          setequal(cbsa$GEOID, counties$GEOID))

panel <- expand_grid(GEOID = counties$GEOID, year = 2006:2025, month = 1:12) |>
  mutate(n_days = as.integer(days_in_month(make_date(year, month, 1)))) |>
  left_join(counties, by = "GEOID", relationship = "many-to-one") |>
  left_join(cbsa, by = "GEOID", relationship = "many-to-one")
stopifnot(nrow(panel) == 3108 * 240)
count(panel, cbsa_type)

Every join in the script goes through this small helper. It does a left_join(), lets dplyr check the declared relationship, and stops if the row count changes.

# All later joins follow one pattern: left_join() onto the panel, declare
# the relationship (dplyr stops with an error if a key repeats where it
# should not), then check that the panel still has the same number of rows.
# A join that adds rows would silently double-count county-months.
add_source <- function(panel, source, by, relationship = "one-to-one") {
  n_before <- nrow(panel)
  out <- left_join(panel, source, by = by, relationship = relationship)
  stopifnot(nrow(out) == n_before)
  out
}

Population (SEER). Key: GEOID + year; many-to-one, because one county-year matches 12 months. No NA: every county has every year, and 2025 is filled from 2024 and flagged.

# SEER July 1 populations run to 2024. For 2025 we hold the 2024 numbers
# flat and flag those rows, so you can drop or treat them differently.
# One county-year matches 12 county-months: a many-to-one join.
pop <- read_csv(
  file.path(clean_dir, "seer_county_year_2006_2024.csv"),
  col_types = cols_only(
    GEOID = col_character(), year = col_integer(), pop_total = col_double(),
    pop_under5 = col_double(), pop_under18 = col_double(),
    pop_65p = col_double()
  )
)
stopifnot(nrow(pop) == 3108 * 19)

pop <- pop |>
  bind_rows(pop |> filter(year == 2024) |> mutate(year = 2025L)) |>
  mutate(pop_carried_forward = year == 2025)

panel <- add_source(panel, pop, by = c("GEOID", "year"),
                    relationship = "many-to-one")
rm(pop)  # free memory: the panel now holds these columns
stopifnot(!anyNA(panel$pop_total))

CSU smoke. Key: GEOID + year + month; one-to-one. No NA: the CSU monthly file is complete. A 0 is a true zero (no resident under a plume on any day); negative monthly means are real and should be kept.

# CSU smoke PM2.5 (median background), already monthly and complete. Every
# month is complete (n_days_obs equals the days in the month), so we check
# that once and drop the bookkeeping columns. rename_with() adds the source
# prefix to every column except the keys.
csu <- read_csv(
  file.path(clean_dir, "csu_smoke_county_month_2006_2025.csv"),
  col_types = cols(GEOID = col_character(), year = col_integer(),
                   month = col_integer(), complete_month = col_logical(),
                   csu_release = col_date(), .default = col_double())
)
stopifnot(nrow(csu) == 3108 * 240, all(csu$complete_month))

csu <- csu |>
  select(-n_days_obs, -complete_month, -csu_release) |>
  rename(total_pm25 = total_pm25_mean) |>
  rename_with(\(x) str_c("csu_", x), -c(GEOID, year, month))

panel <- add_source(panel, csu, by = c("GEOID", "year", "month"))
rm(csu)

HMS plumes. Key: GEOID + year + month; one-to-one. No NA. Counts are days with a plume over the county; hms_analysed_days is the denominator to use for a share of days.

# HMS plume days. The monthly file is complete, and a 0 is a real zero on
# the analysed days. analysed_days is the same for every county; divide by
# it, not by n_days, if you want a share of days.
hms <- read_csv(
  file.path(clean_dir, "hms_county_month_2006_2025.csv"),
  col_types = cols(GEOID = col_character(), year = col_integer(),
                   month = col_integer(), .default = col_double())
) |>
  rename_with(\(x) str_c("hms_", x), -c(GEOID, year, month))
stopifnot(nrow(hms) == 3108 * 240)

panel <- add_source(panel, hms, by = c("GEOID", "year", "month"))
rm(hms)
# The panel already has n_days; check that HMS agrees, then drop its copy.
stopifnot(all(panel$hms_n_days == panel$n_days))
panel <- select(panel, -hms_n_days)

Childs smoke. Key: GEOID + year + month; one-to-one. NA in all of 2024 and 2025, because the Childs file ends in 2023: NA means “not available”, not “no smoke”.

# Childs et al. smoke PM2.5 ends in December 2023, so its columns are NA
# ("not available") in 2024 and 2025. Before then the file is complete and
# 0 means no smoke day in the month.
childs <- read_csv(
  file.path(clean_dir, "childs_smoke_county_month_2006_2023.csv"),
  col_types = cols(GEOID = col_character(), year = col_integer(),
                   month = col_integer(), .default = col_double())
) |>
  rename(smoke_pm25 = smoke_mean) |>
  rename_with(\(x) str_c("childs_", x), -c(GEOID, year, month))
stopifnot(nrow(childs) == 3108 * 216)

panel <- add_source(panel, childs, by = c("GEOID", "year", "month"))
rm(childs)

EPA monitors (AQS). Key: GEOID + year + month; one-to-one. Only 202,737 of the 745,920 county-months have a row. Where a row is missing, the day counts are set to 0 and the means stay NA (“no monitor reading”). A mean is present exactly when its day count is above 0.

# EPA monitors cover only about a third of counties, and the monthly file
# has a row only for county-months with at least one reading. After the
# join, a county-month without a row has NA everywhere. The day counts
# have a true answer there, 0 days measured, so we fill them with 0. The
# means stay NA: no monitor, no value. Never read an NA mean as clean air.
aqs <- read_csv(
  file.path(clean_dir, "aqs_county_month_2006_2025.csv"),
  col_types = cols(GEOID = col_character(), year = col_integer(),
                   month = col_integer(), .default = col_double())
) |>
  rename(o3_mda8_ppb = o3_mda8_mean_ppb) |>
  rename_with(\(x) str_c("aqs_", x), -c(GEOID, year, month))

panel <- add_source(panel, aqs, by = c("GEOID", "year", "month")) |>
  mutate(across(c(aqs_pm25_days, aqs_pm25_days_gt35, aqs_o3_days,
                  aqs_o3_days_gt70),
                \(x) replace_na(x, 0)))
rm(aqs)
# Means are present exactly when at least one day was measured.
stopifnot(all(is.na(panel$aqs_pm25_mean) == (panel$aqs_pm25_days == 0)),
          all(is.na(panel$aqs_o3_mda8_ppb) == (panel$aqs_o3_days == 0)))

nClimGrid weather. Key: GEOID + year + month; one-to-one. No NA. wx_imputed marks Lexington city, Virginia, whose values are copied from the surrounding county.

# nClimGrid station-based weather, complete with no missing cells. The
# prefix wx_ marks it as the weather source of record; ERA5 (next chunk)
# is a second, model-based weather source.
wx <- read_csv(
  file.path(clean_dir, "nclimgrid_county_month_2006_2025.csv"),
  col_types = cols(GEOID = col_character(), year = col_integer(),
                   month = col_integer(), weather_imputed = col_logical(),
                   .default = col_double())
) |>
  rename(imputed = weather_imputed) |>
  rename_with(\(x) str_c("wx_", x), -c(GEOID, year, month))
stopifnot(nrow(wx) == 3108 * 240)

panel <- add_source(panel, wx, by = c("GEOID", "year", "month"))
rm(wx)
stopifnot(all(panel$wx_n_days == panel$n_days))
panel <- select(panel, -wx_n_days)

ERA5. Key: GEOID + year + month; one-to-one. No NA.

# ERA5 reanalysis: surface weather, 10 m winds, and the 500 hPa height.
# Its months are made of UTC days (Coordinated Universal Time, the clock at
# Greenwich), which hardly matters for monthly means.
era5 <- read_csv(
  file.path(clean_dir, "era5_county_month_2006_2025.csv"),
  col_types = cols(GEOID = col_character(), year = col_integer(),
                   month = col_integer(), .default = col_double())
) |>
  rename_with(\(x) str_c("era5_", x), -c(GEOID, year, month))
stopifnot(nrow(era5) == 3108 * 240)

panel <- add_source(panel, era5, by = c("GEOID", "year", "month"))
rm(era5)
stopifnot(all(panel$era5_n_days == panel$n_days))
panel <- select(panel, -era5_n_days)

Canada-wide fire activity (FIRMS). The province-day file is summed to one row per month for all of Canada, by the local solar day of each detection (the calendar day by the sun at the fire). Key: year + month; many-to-one, because every county gets the same national value. NA in 2025, because the FIRMS files in the bundle end in 2024.

# Context, not exposure: how much fire did satellites see in all of Canada
# that month? We add the 13 province-days up to one value per month. FRP
# (fire radiative power) is the heat a fire gives off at the moment a
# satellite passes, in MW; a monthly sum of snapshots is an index of fire
# activity, not an energy total. The same national value goes to every
# county (many-to-one). The FIRMS file ends in 2024, so 2025 is NA.
fire_ca <- read_csv(
  file.path(clean_dir, "firms_canada_province_day_2003_2024.csv"),
  col_types = cols(prov_abbr = col_character(), local_date = col_date(),
                   .default = col_double())
) |>
  mutate(year = as.integer(year(local_date)),
         month = as.integer(month(local_date))) |>
  filter(year >= 2006) |>
  summarize(fire_ca_n_detections = sum(n_detections),
            fire_ca_frp_total_mw = round(sum(frp_total_mw), 1),
            .by = c(year, month))
stopifnot(nrow(fire_ca) == 19 * 12)

panel <- add_source(panel, fire_ca, by = c("year", "month"),
                    relationship = "many-to-one")

The write step checks the grain (one row per county-month, no duplicated key) and a few ranges, then writes the compressed file.

# Final checks on the grain: one row per county-month, all 745,920 of them,
# then a few range checks on values that must not be negative.
panel <- arrange(panel, GEOID, year, month)
stopifnot(
  nrow(panel) == 745920,
  nrow(distinct(panel, GEOID, year, month)) == nrow(panel),
  n_distinct(panel$GEOID) == 3108,
  all(panel$hms_plume_days <= panel$n_days),
  all(panel$csu_hms_days <= panel$n_days),
  all(panel$pop_total > 0),
  all(panel$era5_days_north_wind <= panel$n_days)
)
out_month <- file.path(clean_dir, "wildfire_county_month_panel_2006_2025.csv.gz")
write_csv(panel, out_month, na = "")
ncol(panel)

This chunk lists every column with any NA and checks that only the expected columns have them.

# Where are the NAs? Each NA has one meaning per source: Childs ends in
# 2023, FIRMS ends in 2024, and AQS means are NA where no monitor reported.
# Every other column should be complete.
na_share_month <- panel |>
  summarize(across(everything(), \(x) mean(is.na(x)))) |>
  pivot_longer(everything(), names_to = "column", values_to = "na_share")
na_share_month |>
  filter(na_share > 0) |>
  mutate(na_share = round(na_share, 4))
na_cols <- filter(na_share_month, na_share > 0)$column
stopifnot(all(str_starts(na_cols, "childs_") | str_starts(na_cols, "fire_ca_") |
                na_cols %in% c("aqs_pm25_mean", "aqs_o3_mda8_ppb")))

An example that writes no file: population-weighted CSU smoke by month in the four eastern groups of states (the east_split column). June 2023 stands out in every group.

# Example (writes no file): population-weighted CSU smoke PM2.5 by month in
# the four eastern groups of states. weighted.mean() weights each county by
# its residents, so the line shows what the average resident breathed.
panel |>
  filter(east_split != "Other") |>
  summarize(smoke = weighted.mean(csu_smoke_pm25, pop_total),
            .by = c(east_split, year, month)) |>
  mutate(month_start = make_date(year, month, 1)) |>
  ggplot(aes(month_start, smoke)) +
  geom_line(linewidth = 0.3) +
  facet_wrap(~east_split, ncol = 1) +
  labs(x = NULL, y = "Smoke PM2.5 (ug/m3), population-weighted")

The daily panel starts the same way, from a complete frame of the 3,108 counties and the 365 days of 2023. The monthly panel is removed first to free memory.

# The daily panel covers 2023 only, the year of the big Canadian smoke
# episodes (June 2023 in the East). First free the memory the monthly panel
# used, then build the complete frame: 3,108 counties x 365 days.
rm(panel, fire_ca)

day_panel <- expand_grid(
  GEOID = counties$GEOID,
  date = seq(ymd("2023-01-01"), ymd("2023-12-31"), by = "day")
) |>
  left_join(select(counties, GEOID, county_name, state_abbr, east_split),
            by = "GEOID", relationship = "many-to-one")
stopifnot(nrow(day_panel) == 3108 * 365)

This helper reads only the 2023 rows of a multi-year file, one piece at a time.

# Three daily sources come as one file for all of 2006-2025 (4 to 5.5
# million rows). read_csv_chunked() reads 500,000 rows at a time and keeps
# only the 2023 rows of each piece, so the whole file never sits in memory.
read_2023 <- function(file, col_types) {
  read_csv_chunked(
    file,
    callback = DataFrameCallback$new(\(chunk, pos) filter(chunk, year(date) == 2023)),
    chunk_size = 500000,
    col_types = col_types
  )
}

CSU daily smoke. Key: GEOID + date; one-to-one. No NA. csu_hms_share is the share of residents under a plume; use it, not csu_smoke_pm25 == 0, to find plume-free days.

# CSU smoke and total PM2.5 for 2023, complete. hms_share is the share of
# residents under a plume; smoke_pm25 is exactly 0 when it is 0.
csu_day <- read_csv(
  file.path(clean_dir, "csu_smoke_county_day", "csu_smoke_county_day_2023.csv.gz"),
  col_types = cols_only(GEOID = col_character(), date = col_date(),
                        smoke_pm25 = col_double(), total_pm25 = col_double(),
                        hms_share = col_double())
) |>
  rename_with(\(x) str_c("csu_", x), -c(GEOID, date))
stopifnot(nrow(csu_day) == 3108 * 365)

day_panel <- add_source(day_panel, csu_day, by = c("GEOID", "date"))
rm(csu_day)

HMS daily plumes. Two joins. The calendar joins on date alone (many-to-one: one day, all counties) and adds hms_analysed. The county-day file joins on GEOID + date (one-to-one) but lists only county-days with a plume. On an analysed day without a row, the plume columns become 0 (no plume seen). On the one day of 2023 that HMS did not analyse, 12 May, they stay NA (unknown), for all 3,108 counties.

# The HMS county-day file lists only county-days with a plume. A county-day
# without a row had no plume, which is a real 0 only if HMS analysts worked
# that day. The calendar says which days were analysed. In 2023 every plume
# has a density label, so max_density = 0 can only mean "no plume".
hms_calendar <- read_csv(
  file.path(clean_dir, "hms_calendar_2005_2025.csv"),
  col_types = cols_only(date = col_date(), analysed = col_logical())
) |>
  filter(year(date) == 2023) |>
  rename(hms_analysed = analysed)

hms_day <- read_2023(
  file.path(clean_dir, "hms_county_day_2006_2025.csv.gz"),
  col_types = cols(GEOID = col_character(), date = col_date(),
                   .default = col_double())
) |>
  rename_with(\(x) str_c("hms_", x), -c(GEOID, date))
# Every plume row must fall on an analysed day.
stopifnot(all(hms_day$date %in% hms_calendar$date[hms_calendar$hms_analysed]))

day_panel <- day_panel |>
  add_source(hms_calendar, by = "date", relationship = "many-to-one") |>
  add_source(hms_day, by = c("GEOID", "date")) |>
  mutate(across(c(hms_n_plumes, hms_max_density, hms_pop_share_any,
                  hms_pop_share_medium_plus, hms_pop_share_heavy),
                \(x) if_else(hms_analysed, replace_na(x, 0), NA_real_)))
rm(hms_day)
filter(hms_calendar, !hms_analysed)

EPA daily monitors. Key: GEOID + date; one-to-one. NA in the means where no monitor reported that day (82.5% of county-days for PM2.5); the site counts are 0 there.

# EPA monitor means for the county. No row means no monitor reported that
# day: the means stay NA, and the site counts become 0.
aqs_day <- read_2023(
  file.path(clean_dir, "aqs_county_day_2006_2025.csv.gz"),
  col_types = cols(GEOID = col_character(), date = col_date(),
                   .default = col_double())
) |>
  rename_with(\(x) str_c("aqs_", x), -c(GEOID, date))

day_panel <- add_source(day_panel, aqs_day, by = c("GEOID", "date")) |>
  mutate(across(c(aqs_n_sites_pm25, aqs_n_sites_o3), \(x) replace_na(x, 0)))
rm(aqs_day)
stopifnot(all(is.na(day_panel$aqs_pm25_mean) == (day_panel$aqs_n_sites_pm25 == 0)))

EMBER model. Key: GEOID + date; one-to-one. NA outside the modeled period, 11 April to 29 September 2023 (NA = not modeled).

# EMBER model runs cover 11 April to 29 September 2023. Outside those days
# its columns are NA: not modeled. pm25_fire is PM2.5 from U.S. and Canadian
# fires; pm25_canada_fire is the part from Canadian fires.
ember_day <- read_csv(
  file.path(clean_dir, "ember_county_day_2023.csv"),
  col_types = cols_only(GEOID = col_character(), date = col_date(),
                        pm25_base = col_double(), pm25_fire = col_double(),
                        pm25_canada_fire = col_double())
) |>
  rename_with(\(x) str_c("ember_", x), -c(GEOID, date))
stopifnot(nrow(ember_day) == 3108 * 172)

day_panel <- add_source(day_panel, ember_day, by = c("GEOID", "date"))
rm(ember_day)

nClimGrid daily weather. Key: GEOID + date; one-to-one. No NA.

# nClimGrid daily weather for 2023, complete. imputed is TRUE only for
# Lexington city, Virginia, which copies Rockbridge County.
wx_day <- read_csv(
  file.path(clean_dir, "nclimgrid_county_day", "nclimgrid_county_day_2023.csv.gz"),
  col_types = cols(GEOID = col_character(), date = col_date(),
                   weather_imputed = col_logical(), .default = col_double())
) |>
  rename(imputed = weather_imputed) |>
  rename_with(\(x) str_c("wx_", x), -c(GEOID, date))
stopifnot(nrow(wx_day) == 3108 * 365)

day_panel <- add_source(day_panel, wx_day, by = c("GEOID", "date"))
rm(wx_day)

ERA5 daily winds. Key: GEOID + date; one-to-one. The script adds the wind direction, in degrees clockwise from north that the wind comes from, with the same formula as the ERA5 section. It is NA only on calm days, when both wind parts round to 0 (4 county-days in 2023).

# ERA5 10 m winds. u is the eastward part and v the northward part of the
# wind arrow. Weather reports give the direction the wind comes FROM, in
# degrees clockwise from north (0 = from the north, 90 = from the east).
# This is the same formula as in the ERA5 section; a calm day (u = v = 0)
# has no direction.
wind_from_deg <- function(u, v) {
  if_else(u == 0 & v == 0, NA_real_, (270 - atan2(v, u) * 180 / pi) %% 360)
}

era5_day <- read_csv(
  file.path(clean_dir, "era5_county_day", "era5_county_day_2023.csv.gz"),
  col_types = cols_only(GEOID = col_character(), date = col_date(),
                        u10_ms = col_double(), v10_ms = col_double(),
                        ws10_ms = col_double())
) |>
  mutate(wind_from_deg = round(wind_from_deg(u10_ms, v10_ms), 1)) |>
  rename_with(\(x) str_c("era5_", x), -c(GEOID, date))
stopifnot(nrow(era5_day) == 3108 * 365)

day_panel <- add_source(day_panel, era5_day, by = c("GEOID", "date"))
rm(era5_day)

Childs daily smoke. Key: GEOID + date; one-to-one. No NA: 0 on days that were not smoke days for Childs.

# Childs et al. smoke PM2.5 also covers 2023 and is complete: 0 on days
# that were not smoke days.
childs_day <- read_csv(
  file.path(clean_dir, "childs_smoke_county_day", "childs_smoke_county_day_2023.csv.gz"),
  col_types = cols(GEOID = col_character(), date = col_date(),
                   smoke_pm25 = col_double(), smoke_day = col_logical())
) |>
  rename_with(\(x) str_c("childs_", x), -c(GEOID, date))
stopifnot(nrow(childs_day) == 3108 * 365)

day_panel <- add_source(day_panel, childs_day, by = c("GEOID", "date"))
rm(childs_day)

Nearby-fire Canadian share. Key: GEOID + date; one-to-one. The file has a row only for county-days with positive CSU smoke (448,264 of the 1,134,420 county-days of 2023). Elsewhere the columns are NA, meaning “not applicable: no smoke to attribute”, not “no fire”. Within the covered rows, nearfire_share_ca_sameday is also NA on 86 county-days when no fire burned within 1,500 km that day.

# The nearby-fire Canadian share (Canadian FRP within 1,500 km divided by
# all FRP within 1,500 km) exists only on county-days with CSU smoke above
# 0. Elsewhere it is NA: not applicable, because there was no smoke to
# attribute. The lag version adds FRP over the day and the 3 days before.
fire_share_day <- read_2023(
  file.path(clean_dir, "county_day_fire_share_2006_2025.csv.gz"),
  col_types = cols_only(
    GEOID = col_character(), date = col_date(),
    share_ca_sameday = col_double(), share_ca_lag03 = col_double(),
    ca_dominated_sameday = col_logical(), ca_dominated_lag03 = col_logical()
  )
) |>
  rename_with(\(x) str_c("nearfire_", x), -c(GEOID, date))

day_panel <- add_source(day_panel, fire_share_day, by = c("GEOID", "date"))
rm(fire_share_day)
# A share appears only where CSU smoke was positive (it can still round
# to 0.00 when tiny).
stopifnot(all(day_panel$csu_smoke_pm25[!is.na(day_panel$nearfire_share_ca_lag03)] >= 0))

The daily write step checks the grain and two ranges, then writes the file.

# Grain: one row per county-day of 2023, all 1,134,420 of them.
day_panel <- arrange(day_panel, GEOID, date)
stopifnot(
  nrow(day_panel) == 1134420,
  nrow(distinct(day_panel, GEOID, date)) == nrow(day_panel),
  all(between(day_panel$csu_hms_share, 0, 1)),
  all(day_panel$era5_ws10_ms >= 0)
)
out_day <- file.path(clean_dir, "wildfire_county_day_2023.csv.gz")
write_csv(day_panel, out_day, na = "")
ncol(day_panel)

The NA shares of the daily panel, column by column:

# NA shares in the daily panel. HMS columns are NA on the one 2023 day
# without HMS analysis, EMBER outside its modeled season, AQS where no
# monitor reported, and the fire share where CSU smoke was not positive.
na_share_day <- day_panel |>
  summarize(across(everything(), \(x) mean(is.na(x)))) |>
  pivot_longer(everything(), names_to = "column", values_to = "na_share")
na_share_day |>
  filter(na_share > 0) |>
  mutate(na_share = round(na_share, 4))

An example that writes no file: the ten smokiest county-days of June 2023 by CSU smoke, next to the monitor reading, EMBER’s Canadian-fire PM2.5, the nearby-fire share, and the wind direction. Nine of the ten are in eastern Pennsylvania on 7 June, with winds from the northwest (315°–323°) and a Canadian share above 0.92.

# Example (writes no file): the ten smokiest county-days of June 2023 by
# CSU smoke PM2.5, next to the monitor reading, the EMBER Canadian-fire
# PM2.5, the nearby-fire Canadian share, and the wind direction.
day_panel |>
  filter(month(date) == 6) |>
  slice_max(csu_smoke_pm25, n = 10) |>
  select(date, county_name, state_abbr, csu_smoke_pm25, aqs_pm25_mean,
         ember_pm25_canada_fire, nearfire_share_ca_sameday,
         era5_wind_from_deg) |>
  print(width = Inf)

In a later session, read the panels back with declared column types, one at a time:

# How to read the panels in a later session. Plain read_csv() guesses the
# types correctly here, but declaring them is safer: GEOID must stay text,
# and the logical flags should not depend on a guess. Every other column
# is a number (dbl). Each panel needs several hundred MB of memory, so we
# keep only one in memory at a time.
rm(day_panel)
month_types <- cols(
  GEOID = col_character(), year = col_integer(), month = col_integer(),
  county_name = col_character(), state_abbr = col_character(),
  census_region = col_character(), census_division = col_character(),
  east_split = col_character(), cbsa_type = col_character(),
  pop_carried_forward = col_logical(), wx_imputed = col_logical(),
  .default = col_double()
)
panel <- read_csv(out_month, col_types = month_types)
stopifnot(nrow(panel) == 745920, ncol(panel) == 64, nrow(problems(panel)) == 0)
rm(panel)

day_types <- cols(
  GEOID = col_character(), date = col_date(), county_name = col_character(),
  state_abbr = col_character(), east_split = col_character(),
  hms_analysed = col_logical(), wx_imputed = col_logical(),
  childs_smoke_day = col_logical(),
  nearfire_ca_dominated_sameday = col_logical(),
  nearfire_ca_dominated_lag03 = col_logical(),
  .default = col_double()
)
day_panel <- read_csv(out_day, col_types = day_types)
stopifnot(nrow(day_panel) == 1134420, ncol(day_panel) == 36,
          nrow(problems(day_panel)) == 0)

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
wildfire_county_month_panel_2006_2025.csv.gz	745,920	a county × month, January 2006 – December 2025 (3,108 × 240; complete)	53.4 MB (202.2 MB unzipped)
wildfire_county_day_2023.csv.gz	1,134,420	a county × day of 2023 (3,108 × 365; complete)	39.8 MB (172.6 MB unzipped)

Clean variables

wildfire_county_month_panel_2006_2025.csv.gz: 64 columns, one row per county × month, sorted by GEOID, year, and month; the key is GEOID + year + month. The types are the ones plain read_csv() gives; chunk m12-read-back reads year and month as int. “Source” names the earlier section and, when it differs, the original column name.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (2-digit state + 3-digit county), 2020 boundaries
year	dbl	—	Calendar year, 2006–2025
month	dbl	—	Calendar month, 1–12
n_days	dbl	days	Days in the month (28–31); equals the day counts of HMS, nClimGrid, and ERA5
county_name	chr	—	County name with its legal description (places section)
state_abbr	chr	—	2-letter state abbreviation
census_region	chr	—	Census region: Northeast, Midwest, South, West
census_division	chr	—	One of the 9 Census divisions
east_split	chr	—	The project’s eastern groups: New England, Mid-Atlantic, Great Lakes, Upper Midwest, or Other (places section)
cbsa_type	chr	—	metro (metropolitan area), micro (micropolitan area), or none (in no CBSA; empty in county_cbsa_2023.csv). July 2023 delineation; approximate for the 8 Connecticut counties
pop_total	dbl	persons	SEER July 1 population of the year, all ages. 2025 = 2024
pop_under5	dbl	persons	Ages 0–4 (SEER)
pop_under18	dbl	persons	Ages 0–17 (SEER)
pop_65p	dbl	persons	Ages 65 and older (SEER)
pop_carried_forward	lgl	—	TRUE in 2025, where the SEER 2024 populations are repeated; FALSE otherwise
csu_smoke_pm25	dbl	µg/m³	CSU: monthly mean of daily smoke PM2.5 (median background, population-weighted). Can be negative; keep it. CSU smoke_pm25
csu_total_pm25	dbl	µg/m³	CSU: monthly mean of daily total PM2.5. CSU total_pm25_mean
csu_smoke_max	dbl	µg/m³	CSU: highest daily smoke PM2.5 in the month
csu_hms_days	dbl	days	CSU: days with more than half of residents under an HMS plume (hms_share > 0.5)
csu_days_gt5	dbl	days	CSU: days with smoke PM2.5 above 5 µg/m³
csu_days_gt10	dbl	days	CSU: days above 10 µg/m³
csu_days_gt15	dbl	days	CSU: days above 15 µg/m³
csu_days_gt35	dbl	days	CSU: days above 35 µg/m³
hms_plume_days	dbl	days	HMS: days with at least one plume over the county’s land. HMS plume_days
hms_medium_plus_days	dbl	days	HMS: days whose densest plume was Medium or Heavy. Comparable over time only from June 2010, when labels became complete
hms_heavy_days	dbl	days	HMS: days whose densest plume was Heavy
hms_pop_weighted_plume_days	dbl	days	HMS: plume days weighted by the share of residents under the plume
hms_analysed_days	dbl	days	HMS: days of the month with any plume anywhere (analysed days); the same for every county. Use it as the denominator of a share of days
childs_smoke_pm25	dbl	µg/m³	Childs (v2.0 beta): mean daily smoke PM2.5 over all days of the month, zeros included. Childs smoke_mean. NA in 2024–2025 (not available)
childs_smoke_max	dbl	µg/m³	Childs: highest daily smoke PM2.5. NA in 2024–2025
childs_smoke_days	dbl	days	Childs: smoke days (county-days in the raw Childs file). NA in 2024–2025
childs_days_gt5	dbl	days	Childs: days above 5 µg/m³. NA in 2024–2025
childs_days_gt15	dbl	days	Childs: days above 15 µg/m³. NA in 2024–2025
childs_days_gt35	dbl	days	Childs: days above 35 µg/m³. NA in 2024–2025
aqs_pm25_mean	dbl	µg/m³	AQS: mean of the county’s daily monitor PM2.5 over days with a reading. NA where no PM2.5 monitor reported in the month (81.1% of rows)
aqs_pm25_days	dbl	days	AQS: days with a PM2.5 reading; 0 where the source has no row
aqs_pm25_days_gt35	dbl	days	AQS: days with county PM2.5 above 35.4 µg/m³; 0 where the source has no row
aqs_o3_mda8_ppb	dbl	ppb	AQS: mean daily maximum 8-hour ozone over days with a reading. AQS o3_mda8_mean_ppb. NA where no ozone monitor reported (78.7% of rows)
aqs_o3_days	dbl	days	AQS: days with an ozone reading; 0 where the source has no row
aqs_o3_days_gt70	dbl	days	AQS: days with ozone above 70 ppb; 0 where the source has no row
wx_tmax_mean_c	dbl	°C	nClimGrid: mean daily maximum temperature
wx_tmin_mean_c	dbl	°C	nClimGrid: mean daily minimum temperature
wx_tavg_mean_c	dbl	°C	nClimGrid: mean of (max + min) / 2
wx_prcp_total_mm	dbl	mm	nClimGrid: total precipitation
wx_days_tmax_ge32	dbl	days	nClimGrid: days with maximum temperature ≥ 32 °C
wx_days_tmin_le0	dbl	days	nClimGrid: days with minimum temperature ≤ 0 °C
wx_days_prcp_0	dbl	days	nClimGrid: dry days (county precipitation rounds to 0.00 mm)
wx_days_prcp_0_5	dbl	days	nClimGrid: days with more than 0 and at most 5 mm
wx_days_prcp_5_15	dbl	days	nClimGrid: days with more than 5 and at most 15 mm
wx_days_prcp_gt15	dbl	days	nClimGrid: days with more than 15 mm
wx_imputed	lgl	—	nClimGrid: TRUE for Lexington city, Virginia (51678), copied from Rockbridge County. nClimGrid weather_imputed
era5_t2m_c	dbl	°C	ERA5: mean air temperature 2 m above the ground (UTC days)
era5_d2m_c	dbl	°C	ERA5: mean dewpoint temperature at 2 m
era5_rh_pct	dbl	%	ERA5: mean relative humidity
era5_u10_ms	dbl	m/s	ERA5: mean eastward wind at 10 m (negative = toward the west)
era5_v10_ms	dbl	m/s	ERA5: mean northward wind at 10 m (negative = toward the south)
era5_ws10_ms	dbl	m/s	ERA5: mean of the daily wind speeds
era5_blh_m	dbl	m	ERA5: mean boundary-layer height (depth of the well-mixed air near the ground)
era5_ssrd_wm2	dbl	W/m²	ERA5: mean sunlight reaching the ground, nights included
era5_z500_gpm	dbl	gpm	ERA5: mean height of the 500 hPa pressure level
era5_z500_anom_gpm	dbl	gpm	ERA5: mean 500 hPa height minus the county’s 1991–2005 normal; drifts upward over time
era5_days_north_wind	dbl	days	ERA5: days whose daily-mean wind came from 315° through 45° (northwest to northeast)
fire_ca_n_detections	dbl	count	FIRMS: MODIS vegetation-fire detections in all of Canada that month (local solar days), the same for every county. NA in 2025
fire_ca_frp_total_mw	dbl	MW	FIRMS: sum of fire radiative power (FRP) over those detections, 1 decimal. A sum of satellite snapshots, an index of fire activity, not an energy total. NA in 2025

wildfire_county_day_2023.csv.gz: 36 columns, one row per county × day of 2023, sorted by GEOID and date; the key is GEOID + date. The types are the ones plain read_csv() gives.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code, 2020 boundaries
date	date	—	Day, 2023-01-01 to 2023-12-31. Each source dates its days its own way (see “Checks and cautions”)
county_name	chr	—	County name (places section)
state_abbr	chr	—	2-letter state abbreviation
east_split	chr	—	The project’s eastern groups, or Other
csu_smoke_pm25	dbl	µg/m³	CSU: daily smoke PM2.5 (median background, population-weighted). Exactly 0 when no resident was under a plume; can be negative
csu_total_pm25	dbl	µg/m³	CSU: daily total PM2.5 (kriged from EPA monitors)
csu_hms_share	dbl	share (0–1)	CSU: share of residents in grid cells under an HMS plume. CSU hms_share
hms_analysed	lgl	—	HMS calendar: TRUE if HMS drew at least one plume anywhere that day; FALSE only on 12 May 2023. HMS analysed
hms_n_plumes	dbl	count	HMS: plume polygons touching the county’s land. 0 on analysed days without a plume; NA on 12 May 2023
hms_max_density	dbl	code	HMS: densest plume, 1 = Light, 2 = Medium, 3 = Heavy; 0 = no plume (in 2023 every plume has a label). NA on 12 May 2023
hms_pop_share_any	dbl	share (0–1)	HMS: share of the county’s 2020 population whose block-group center lies inside any plume. 0 / NA as for hms_n_plumes
hms_pop_share_medium_plus	dbl	share (0–1)	HMS: the same for Medium or Heavy plumes
hms_pop_share_heavy	dbl	share (0–1)	HMS: the same for Heavy plumes
aqs_pm25_mean	dbl	µg/m³	AQS: mean daily PM2.5 over the county’s monitors that reported. NA where none reported (82.5% of rows)
aqs_n_sites_pm25	dbl	count	AQS: number of PM2.5 sites in the mean; 0 where none reported
aqs_o3_mda8_ppb	dbl	ppb	AQS: mean daily maximum 8-hour ozone over the county’s sites. NA where none reported (79.4% of rows)
aqs_n_sites_o3	dbl	count	AQS: number of ozone sites; 0 where none reported
ember_pm25_base	dbl	µg/m³	EMBER: modeled PM2.5 with all emissions. NA outside 11 April – 29 September 2023 (52.9% of rows)
ember_pm25_fire	dbl	µg/m³	EMBER: PM2.5 from U.S. and Canadian fires (base run minus no-fire run). NA outside the modeled period
ember_pm25_canada_fire	dbl	µg/m³	EMBER: PM2.5 from Canadian fires (base run minus no-Canadian-fire run). NA outside the modeled period
wx_tmax_c	dbl	°C	nClimGrid: daily maximum temperature
wx_tmin_c	dbl	°C	nClimGrid: daily minimum temperature
wx_tavg_c	dbl	°C	nClimGrid: (max + min) / 2
wx_prcp_mm	dbl	mm	nClimGrid: daily precipitation
wx_imputed	lgl	—	nClimGrid: TRUE for Lexington city, Virginia
era5_u10_ms	dbl	m/s	ERA5: daily-mean eastward wind at 10 m (UTC day)
era5_v10_ms	dbl	m/s	ERA5: daily-mean northward wind at 10 m
era5_ws10_ms	dbl	m/s	ERA5: speed of the daily-mean wind
era5_wind_from_deg	dbl	degrees	Direction the daily-mean wind comes from, clockwise from north (0 = from the north, 90 = from the east), computed here from the two wind columns; 1 decimal. NA on calm days (4 rows)
childs_smoke_pm25	dbl	µg/m³	Childs: daily smoke PM2.5; 0 on non-smoke days
childs_smoke_day	lgl	—	Childs: TRUE if the county-day is a smoke day for Childs
nearfire_share_ca_sameday	dbl	share (0–1)	Episodes section: Canadian FRP ÷ all FRP within 1,500 km, same local solar day. NA where CSU smoke was not positive (not applicable), and on 86 smoky county-days with no fire within 1,500 km. share_ca_sameday
nearfire_share_ca_lag03	dbl	share (0–1)	The same with FRP summed over the day and the 3 days before. NA where CSU smoke was not positive. share_ca_lag03
nearfire_ca_dominated_sameday	lgl	—	TRUE when the unrounded same-day share is at least 0.5; NA where CSU smoke was not positive. ca_dominated_sameday
nearfire_ca_dominated_lag03	lgl	—	The same for the lag share. ca_dominated_lag03

Checks and cautions

• Every value equals its source. R/check_m12.R compares all 92 joined columns of both panels with the clean files they came from, over every row, not only a sample: there are 0 mismatches and a largest difference of 0. For each source it also prints a spot check of 5 random keys, with the panel value next to the source value (logs/check_m12.log). The fills behave as described: the AQS day counts are 0 exactly where the source has no row, the HMS columns are 0 on analysed days without a plume (the 532,963 plume rows of 2023 are exactly the panel rows with hms_n_plumes > 0), and era5_wind_from_deg equals an independent formula (the bearing the wind blows toward, plus 180°) on every county-day.
• Agreement with the research project. HMS plume days match the project’s monthly HMS table in 745,606 of 745,920 county-months (314 differ by 1 day, as the HMS section explains). Childs monthly smoke agrees within 0.005 µg/m³ (rounding), with identical smoke-day counts, and SEER populations are identical in all 59,052 county-years.
• Complete panels. Both files have every county on every month or day, with no duplicated key. Missing values (NA) appear only where a source does not apply or was not observed. Monthly panel: Childs columns 10.0% (2024–2025), aqs_pm25_mean 81.1%, aqs_o3_mda8_ppb 78.7%, fire_ca_ columns 5.0% (2025); all other columns 0. Daily panel: HMS plume columns 0.27% (12 May), aqs_pm25_mean 82.5%, aqs_o3_mda8_ppb 79.4%, EMBER 52.9%, nearfire_ columns 60.5%, era5_wind_from_deg 4 rows; all other columns 0.
• An NA is not a zero. NA means “not observed” for AQS and for HMS on 12 May 2023, “not available” for Childs after 2023, FIRMS in 2025, and EMBER outside its season, and “not applicable” for the nearby-fire share on days without CSU smoke. Do not replace these NAs with 0. If you average a column with NAs, mean(x, na.rm = TRUE) averages only the counties and months that have a value, which for AQS is a different, mostly urban set of counties.
• Monitors cover few counties. Only 140,915 county-months (18.9%) have a PM2.5 mean and 158,957 (21.3%) an ozone mean; in 2023 only 198,112 county-days (17.5%) have a PM2.5 reading. A regression on aqs_pm25_mean uses a small, urban sample; CSU total PM2.5 covers every county but is interpolated from those same monitors.
• Several smoke measures, not one. CSU, Childs, and EMBER estimate surface smoke PM2.5 in three different ways, HMS shows smoke anywhere in the air column, and the monitors measure total PM2.5. They do not agree day by day. On 28 June 2023, De Witt County, Illinois, has 197 µg/m³ of CSU smoke and 140 µg/m³ of EMBER Canadian-fire PM2.5, but a nearby-fire Canadian share of only 0.41: the share counts fires within 1,500 km that day, not the fires the smoke came from. Pick one measure for your main analysis and use the others as checks.
• Days are not all the same day. CSU, AQS, Childs, and nClimGrid use local days (nClimGrid’s are observers’ reporting days); HMS and ERA5 use UTC days, which start 4–8 hours before local midnight in the U.S.; the nearby-fire share uses fires’ local solar days; and EMBER’s day is probably local standard time (see its section). A same-date join can be off by part of a day, which matters for single days but hardly for months.
• Choices carried over from the sources. CSU smoke uses the median background (the CSU section explains the choice; the mean background gives lower smoke). HMS density labels are incomplete before June 2010. Childs is a beta version. AQS county means include both original and corrected T640 monitor data. Lexington city’s weather is a copy of Rockbridge County’s (wx_imputed). The 2024 and 2025 CSU years come from later releases than 2006–2023.
• 2025 population is held flat. pop_carried_forward marks the 2025 rows. Weighted averages for 2025 use 2024 populations.
• Canadian fire activity is a national total. fire_ca_frp_total_mw is the same for every county in a month and peaks in September 2023 (8.40 million MW from 46,810 detections) and June 2023 (8.08 million MW from 53,144 detections). It says how much fire Canada had, not how much smoke reached a county. In a regression with month or year-by-month fixed effects it is absorbed completely.
• Eastern smoke peaked in June 2023. In the example chunk, population-weighted CSU smoke reaches 19.9 µg/m³ in the Great Lakes group, 17.2 in the Mid-Atlantic group, and 14.1 in the Upper Midwest group in June 2023, the three highest group-months of 2006–2025; the next is the Upper Midwest in July 2021 (12.4).
• Reading the files. Plain read_csv() reads both files without problems: GEOID stays text because the first county, 01001, has a leading zero, and the logical columns are guessed correctly. Chunk m12-read-back declares the types anyway. In R the monthly panel takes about 360 MB of memory and the daily panel about 290 MB; keep only one in memory if your computer has 8 GB or less.
• Memory and time. The whole script runs in about 30 seconds on the instructor’s Mac, with a peak memory use of about 2 GB (the operating system’s peak “memory footprint” was 2.0 GB). It removes each source after joining it and reads the long daily files in 500,000-row pieces to stay there. When you run it with Rscript from the Terminal instead of in RStudio, R saves the example plot to Rplots.pdf in your project folder; you can delete it.

Use, licences, and citation

• All sources are free public data. U.S. federal data (Census, NOAA, NASA, EPA, NCI SEER) are public domain; Statistics Canada boundaries and CWFIS NBAC are under the Open Government Licence – Canada; the CSU product (Dryad) is CC0; EMBER (Zenodo) is CC BY 4.0. Each section gives the citation to use.
• Cite the original provider, not this course, when you use a dataset in your paper.
• Do not add mortality or other health-outcome data to this folder or to your project repository.
• Keep wildfire_data/ out of Git: add wildfire_data/ to your .gitignore. Share code, not data.

Appendix: how the instructor exported project-built files

Three clean datasets (CSU smoke for all years, ERA5 weather, and the smoke episodes) come from raw files too large to share or from the research project’s own pipeline. Students do not run this code; it is here so every clean file has a documented origin.

CSU kriged smoke PM2.5: instructor export

The raw CSU product is 40 NetCDF files of about 515 MB each (20.6 GB in all), too large for the student bundle. This export gives students the county results for every year instead. It reads the research project’s county panels, which the project’s script 05g_csu.R built from all 20 yearly median-background files with the same method as the student section (block-group population weights, nearest cell center, 25 km cut). It keeps the population-weighted, median-background values, renames the columns, rounds them, and writes one CSV file per year plus one monthly file.

Students do not run this script: it needs the research project folder, which is not in the bundle. They use its output files, which are documented in the CSU section and below. The script is R/m07x_instructor_export.R. Run it from the bundle root with the project folder next to the bundle (../wildfire-claude), or set the environment variable WILDFIRE_PROJECT_DIR to another copy. Its setup chunk loads tidyverse and arrow; the chunks below need both. The validation script R/check_m07.R compares the export with the project files. The export’s settings (background, weighting, years) are in the chunk m07x-paths, so a mean-background version needs one changed line. The output file names do not record the background, so such a run would overwrite the median files; write it to another folder.

Provider	Research project build (scripts/R/05_exposure_panel/05g_csu.R) from the CSU Dryad files (version 8) by Pierce, Ford, O’Dell, McGinnis, and Fischer (Colorado State University, its CIRA institute, and the U.S. Department of Energy; see the CSU section)
Coverage	2006–2025, daily and monthly · the 3,108 contiguous-U.S. counties (DC included)
Raw unit	one Parquet row = a weighting × county × day (daily panel, one file per background × year) or a weighting × background × county × month (monthly panel)
Clean unit	one row = a county × day (one file per year) or a county × month
Licence and citation	CC0 1.0; cite the dataset and O’Dell et al. (2019) as in the CSU section
Documentation	CSU section of this guide; Dryad dataset page and README; project codebook data/cleaned/codebook.csv

Raw files

These are the inputs, in the research project folder (default ../wildfire-claude, next to the bundle). They are not in the bundle.

File (in the project’s data/cleaned/panel/)	Size	Format	One row is
csu_county_day/background=median/year=<year>/part-0.parquet, 20 files for 2006–2025	23.8–32.1 MB each; 528.6 MB in all	Apache Parquet (a compressed column-based table format) in a “hive” folder layout: the folder names hold the partition keys background and year	a weighting × county × day: 2 × 1,134,420 rows per year (2 × 1,137,528 in leap years)
csu_county_month.parquet	96.6 MB	Apache Parquet	a weighting × background × county × month: 2 × 2 × 3,108 × 240 = 2,983,680 rows

Raw variables

The daily Parquet files (csu_county_day/background=median/year=<year>/part-0.parquet) have these columns; background and year are folder names, not columns.

Variable	Type	Description
weight_basis	string	population: weights from 2020 block-group centers of population, as in the student section (exported). area_land: equal weight on every cell whose center lies in the county’s land area; a county with no cell center inside gets the cell nearest its internal point (not exported)
GEOID	string	5-character county FIPS code, 2020 boundaries
date	date32	Day of the 24-hour average
smoke	double	Smoke PM2.5, µg/m³: weighted county mean of HMS_Smoke × (PM25 − Background_PM25); negatives kept
total	double	Total kriged PM2.5, µg/m³ (weighted mean of PM25)
background_pm25	double	Seasonal non-smoke background, µg/m³ (weighted mean of Background_PM25)
hms_share	double	Weighted share of the county under an HMS smoke plume, 0 to 1
csu_background	string	Background definition of the source file: median in these folders
csu_release	string	Date Dryad published the source NetCDF: 2025-03-21 (2006–2023), 2026-02-25 (2024), 2026-08-06 (2025). The NetCDF records an earlier creation date in its description attribute: 2025-01-13, 2025-10-22, and 2026-07-30
source_file	string	Name of the source NetCDF, e.g., krigedPM25_2023_v2_statfix_medianbk.nc (the project codebook mislabels it “Raw FIRMS file”)

The monthly Parquet file csu_county_month.parquet has these columns. The project computed every monthly value from unrounded daily values.

Variable	Type	Description
weight_basis	string	population (exported) or area_land, as above
csu_background	string	median (exported) or mean
csu_release	string	Date Dryad published the source NetCDF, as above
GEOID	string	5-character county FIPS code
month_start	date32	First day of the month
S	double	Monthly mean of daily smoke PM2.5 over observed days, µg/m³ (the project’s monthly smoke measure)
total_mean	double	Monthly mean of daily total PM2.5, µg/m³
n_days_obs	int32	Days with a value in the month
smoke_max	double	Highest daily smoke PM2.5 in the month, µg/m³
hms_days	int32	Days with hms_share > 0.5
days_gt5	int32	Days with smoke PM2.5 > 5 µg/m³
days_gt10	int32	Days with smoke PM2.5 > 10 µg/m³
days_gt15	int32	Days with smoke PM2.5 > 15 µg/m³
days_gt35	int32	Days with smoke PM2.5 > 35 µg/m³
share_days_obs	double	n_days_obs / days in the month (1 in every row; not exported)
complete_month	bool	TRUE if every day of the month has a value

Cleaning steps

The script R/m07x_instructor_export.R makes these decisions:

1. Median background, population weights. It keeps the population-weighted, median-background values, the same choices as the student sample.
2. Student column names. smoke becomes smoke_pm25, total becomes total_pm25, S becomes smoke_pm25, and total_mean becomes total_pm25_mean. In the monthly file, the project’s month_start date becomes two integer columns, year and month, the keys of the other monthly files in the bundle.
3. Rounding. PM2.5 values are rounded to 2 decimals and hms_share to 3, as in the student sample.
4. Releases. Daily files list the release by year in the table below; the monthly file keeps a csu_release column.
5. Checks before writing. Every file must be complete (all 3,108 counties on every day or month), have unique keys and no missing values, and come from a single source file per year.

The first chunk sets the paths and the export settings.

# INSTRUCTOR ONLY. The 2006-2025 CSU files are 20 GB, so students get the
# research project's county panels instead of the raw grids. The project
# folder sits next to the bundle; set WILDFIRE_PROJECT_DIR to use another copy.
project_dir <- Sys.getenv("WILDFIRE_PROJECT_DIR", unset = "../wildfire-claude")
panel_dir <- file.path(project_dir, "data", "cleaned", "panel")
stopifnot(dir.exists(file.path(panel_dir, "csu_county_day")))

# Read Parquet single-threaded, as the project build does: its build log
# recorded garbled rows from a threaded arrow read (arrow 24.0.0, 2026-09-29).
options(arrow.use_threads = FALSE)

# Export settings: the background definition (the project built "median" and
# "mean") and the spatial weighting (the project built "population" and
# "area_land").
csu_background <- "median"
csu_weight_basis <- "population"
csu_years <- 2006:2025

day_dir <- file.path(clean_dir, "csu_smoke_county_day")
dir.create(day_dir, recursive = TRUE, showWarnings = FALSE)

The county frame from module m00 defines the complete set of counties.

# The county frame written by module m00: 3,108 contiguous counties plus DC.
counties <- read_csv(file.path(clean_dir, "counties_2020.csv"),
                     col_types = cols_only(GEOID = col_character()))
stopifnot(nrow(counties) == 3108, !anyDuplicated(counties$GEOID))

This function exports one year: it reads the year’s Parquet file, keeps one weighting, renames and rounds, checks the grain, and writes the CSV.

# Export one year of county-day values. The project stores one Parquet file
# per background x year (a "hive" folder layout: background=median/year=2023/),
# holding both weightings; we keep one weighting and rename the columns to
# the student names.
export_csu_year <- function(year) {
  part_file <- file.path(panel_dir, "csu_county_day",
                         str_c("background=", csu_background), str_c("year=", year),
                         "part-0.parquet")
  # Select and filter inside arrow, so only the rows we keep become an R table.
  part <- read_parquet(part_file, as_data_frame = FALSE) |>
    filter(weight_basis == csu_weight_basis) |>
    select(GEOID, date, smoke, total, background_pm25, hms_share, csu_release, source_file) |>
    collect()

  out <- part |>
    transmute(
      GEOID,
      date = as.Date(date),
      smoke_pm25 = round(smoke, 2),
      total_pm25 = round(total, 2),
      background_pm25 = round(background_pm25, 2),
      hms_share = round(hms_share, 3)
    ) |>
    arrange(GEOID, date)

  # Grain: one row per county x day of the year, complete, no missing values.
  year_days <- seq(as.Date(str_c(year, "-01-01")), as.Date(str_c(year, "-12-31")), by = "day")
  stopifnot(
    nrow(out) == nrow(counties) * length(year_days),
    !anyDuplicated(select(out, GEOID, date)),
    setequal(out$GEOID, counties$GEOID),
    setequal(out$date, year_days),
    !anyNA(out),
    all(between(out$hms_share, 0, 1)),
    n_distinct(part$csu_release) == 1, n_distinct(part$source_file) == 1
  )

  out_file <- file.path(day_dir, str_c("csu_smoke_county_day_", year, ".csv.gz"))
  write_csv(out, out_file, na = "")

  # One summary row per year, for the guide's table of releases.
  year_summary <- tibble(
    year = year, rows = nrow(out), csu_release = part$csu_release[1],
    source_file = part$source_file[1], mean_smoke_pm25 = round(mean(out$smoke_pm25), 3),
    size_mb = round(file.size(out_file) / 1e6, 1)
  )
  # Free this year's tables before the next year: without gc(), R holds on to
  # them and peak memory climbs from about 1 GB to over 2 GB across 20 years.
  rm(part, out)
  invisible(gc())
  year_summary
}

Run it for all 20 years and print a summary row per year.

# One year at a time: about 1.1 million county-days reach R per year.
day_summary <- map(csu_years, export_csu_year) |> list_rbind()
stopifnot(nrow(day_summary) == length(csu_years))
print(day_summary, n = Inf)

The monthly file comes from the project’s monthly panel.

# County-month panel: the project's monthly means of daily smoke for the same
# background and weighting. open_dataset() scans the file in pieces, and the
# filter runs inside arrow, so only the 745,920 rows we need (of 2,983,680)
# reach R.
month_raw <- open_dataset(file.path(panel_dir, "csu_county_month.parquet")) |>
  filter(weight_basis == csu_weight_basis, csu_background == .env$csu_background) |>
  select(GEOID, month_start, S, total_mean, smoke_max, hms_days, starts_with("days_gt"),
         n_days_obs, complete_month, csu_release) |>
  collect()

# Integer year and month columns, the keys of the other monthly files in the
# bundle, replace the project's month_start date.
csu_month <- month_raw |>
  transmute(
    GEOID,
    year = as.integer(year(month_start)),
    month = as.integer(month(month_start)),
    smoke_pm25 = round(S, 2),
    total_pm25_mean = round(total_mean, 2),
    smoke_max = round(smoke_max, 2),
    hms_days, days_gt5, days_gt10, days_gt15, days_gt35,
    n_days_obs, complete_month, csu_release
  ) |>
  arrange(GEOID, year, month)

Check the monthly grain and the logic of the counts, then write the file and count rows by release.

# Grain: one row per county x month, 3,108 x 240, complete, no missing values;
# every month has all its days observed.
months <- seq(as.Date("2006-01-01"), as.Date("2025-12-01"), by = "month")
month_dates <- make_date(csu_month$year, csu_month$month)
stopifnot(
  nrow(csu_month) == nrow(counties) * length(months),
  !anyDuplicated(select(csu_month, GEOID, year, month)),
  setequal(csu_month$GEOID, counties$GEOID),
  setequal(month_dates, months),
  !anyNA(csu_month),
  all(csu_month$n_days_obs == days_in_month(month_dates)),
  all(csu_month$complete_month),
  all(csu_month$hms_days <= csu_month$n_days_obs),
  all(csu_month$days_gt35 <= csu_month$days_gt15),
  all(csu_month$days_gt15 <= csu_month$days_gt10),
  all(csu_month$days_gt10 <= csu_month$days_gt5)
)

write_csv(csu_month, file.path(clean_dir, "csu_smoke_county_month_2006_2025.csv"), na = "")
count(csu_month, csu_release)

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
csu_smoke_county_day/csu_smoke_county_day_<year>.csv.gz, 20 files for 2006–2025	1,134,420 per year (1,137,528 in 2008, 2012, 2016, 2020, 2024); 22,703,940 in all	a county × day	5.9–7.1 MB each (about 37 MB uncompressed); 124.8 MB in all
csu_smoke_county_month_2006_2025.csv	745,920	a county × month (3,108 × 240)	41.07 MB

CSU release, source file, and mean county-day smoke PM2.5 by year (population-weighted, median background):

Year	Rows	CSU release	Source NetCDF	Mean smoke_pm25 (µg/m³)	Size
2006	1,134,420	2025-03-21	krigedPM25_2006_v2_statfix_medianbk.nc	0.160	5.93 MB
2007	1,134,420	2025-03-21	krigedPM25_2007_v2_statfix_medianbk.nc	0.479	6.13 MB
2008	1,137,528	2025-03-21	krigedPM25_2008_v2_statfix_medianbk.nc	0.215	6.00 MB
2009	1,134,420	2025-03-21	krigedPM25_2009_v2_statfix_medianbk.nc	0.114	5.86 MB
2010	1,134,420	2025-03-21	krigedPM25_2010_v2_statfix_medianbk.nc	0.181	5.98 MB
2011	1,134,420	2025-03-21	krigedPM25_2011_v2_statfix_medianbk.nc	0.565	6.24 MB
2012	1,137,528	2025-03-21	krigedPM25_2012_v2_statfix_medianbk.nc	0.472	6.32 MB
2013	1,134,420	2025-03-21	krigedPM25_2013_v2_statfix_medianbk.nc	0.364	6.06 MB
2014	1,134,420	2025-03-21	krigedPM25_2014_v2_statfix_medianbk.nc	0.185	5.91 MB
2015	1,134,420	2025-03-21	krigedPM25_2015_v2_statfix_medianbk.nc	0.347	6.00 MB
2016	1,137,528	2025-03-21	krigedPM25_2016_v2_statfix_medianbk.nc	0.171	5.91 MB
2017	1,134,420	2025-03-21	krigedPM25_2017_v2_statfix_medianbk.nc	0.327	6.05 MB
2018	1,134,420	2025-03-21	krigedPM25_2018_v2_statfix_medianbk.nc	0.498	6.11 MB
2019	1,134,420	2025-03-21	krigedPM25_2019_v2_statfix_medianbk.nc	0.233	6.08 MB
2020	1,137,528	2025-03-21	krigedPM25_2020_v2_statfix_medianbk.nc	0.544	6.13 MB
2021	1,134,420	2025-03-21	krigedPM25_2021_v2_statfix_medianbk.nc	1.354	6.62 MB
2022	1,134,420	2025-03-21	krigedPM25_2022_v2_statfix_medianbk.nc	1.045	6.83 MB
2023	1,134,420	2025-03-21	krigedPM25_2023_v2_statfix_medianbk.nc	2.390	7.08 MB
2024	1,137,528	2026-02-25	krigedPM25_2024_v2_statfix_medianbk.nc	1.272	6.90 MB
2025	1,134,420	2026-08-06	krigedPM25_2025_v2_statfix_medianbk.nc	1.072	6.70 MB

Clean variables

The yearly files csu_smoke_county_day_<year>.csv.gz share these columns with the student sample csu_sample_county_day_2023.csv.gz. Rows are sorted by GEOID and date, and no cell is missing.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code (state + county), 2020 boundaries; joins to counties_2020.csv
date	date	—	Day of the 24-hour average (from the raw doy)
smoke_pm25	dbl	µg/m³	Smoke PM2.5: population-weighted mean over the county’s cells of HMS_Smoke × (PM25 − Background_PM25). Exactly 0 when no resident lives under a plume (hms_share = 0): a true zero, not a missing value. The reverse does not hold: it can also be 0.00 on plume days (1,103 county-days in 2023). Because hms_share is rounded to 3 decimals, 7 county-days in 2006–2025 show hms_share = 0 with smoke_pm25 = 0.01. Can be negative; keep those values. 2 decimals
total_pm25	dbl	µg/m³	Total PM2.5: population-weighted mean of the kriged PM25, smoke and non-smoke together. Rarely negative, where kriging undershoots (516 county-days in 2023; 217 to 2,478 per year). 2 decimals
background_pm25	dbl	µg/m³	Non-smoke background: population-weighted mean of the seasonal median Background_PM25. Constant within each quarter, so it changes only on January 1, April 1, July 1, and October 1. Positive except in July–September 2016 in 11 counties of North Dakota and Minnesota (minimum −0.69) and July–September 2022 in Douglas County, Wisconsin (−0.35): kriged PM2.5 can be negative, and so can its median. 2 decimals
hms_share	dbl	share, 0 to 1	Share of the county’s 2020 population living in grid cells under an HMS smoke plume that day (population-weighted mean of HMS_Smoke). 3 decimals, so a share below 0.0005 shows as 0 (31 to 70 county-days per year). The project calls a county-day a smoke day when hms_share > 0.5

csu_smoke_county_month_2006_2025.csv: one row per county × month, sorted by GEOID, year, and month, no missing cells. read_csv() reads year, month, and the counts as dbl and csu_release as a date.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code, 2020 boundaries
year	dbl	—	Calendar year, 2006 to 2025 (from the project’s month_start)
month	dbl	—	Calendar month, 1 (January) to 12 (December) (from month_start)
smoke_pm25	dbl	µg/m³	Monthly mean of daily smoke PM2.5 (the project’s monthly smoke measure, S). 2 decimals
total_pm25_mean	dbl	µg/m³	Monthly mean of daily total PM2.5. 2 decimals
smoke_max	dbl	µg/m³	Highest daily smoke PM2.5 in the month. 2 decimals
hms_days	dbl	days	Days with hms_share > 0.5 (most residents under a plume)
days_gt5	dbl	days	Days with daily smoke PM2.5 > 5 µg/m³
days_gt10	dbl	days	Days with daily smoke PM2.5 > 10 µg/m³
days_gt15	dbl	days	Days with daily smoke PM2.5 > 15 µg/m³
days_gt35	dbl	days	Days with daily smoke PM2.5 > 35 µg/m³ (35 µg/m³ is the EPA 24-hour standard for total PM2.5; here only the smoke part is counted)
n_days_obs	dbl	days	Days in the month with a value; equals the number of days in the month in every row
complete_month	lgl	—	TRUE when every day of the month has a value; TRUE in every row
csu_release	date	—	Release: the date Dryad published the file behind the month: 2025-03-21 for 2006–2023, 2026-02-25 for 2024, 2026-08-06 for 2025. This is not the date CSU made the file: the NetCDF records its creation date in its description attribute (2025-01-13 for 2006–2023, 2025-10-22 for 2024, 2026-07-30 for 2025)

Checks and cautions

• Exact copy of the project panels, up to rounding. R/check_m07.R compares every exported value with the project’s Parquet files. All 20 daily files have the expected rows (3,108 counties × every day of the year), no duplicated GEOID–date pairs, and no missing values. The largest difference from the project is 0.005 µg/m³ for the PM2.5 columns and 0.0005 for hms_share, which is rounding. In the monthly file, the PM2.5 columns also differ by at most 0.005, and every count column and csu_release is identical.
• Same numbers as the student code. The exported 2023 file is identical, cell by cell, to csu_sample_county_day_2023.csv.gz, which the student code builds from the raw 2023 NetCDF.
• Zeros. In the project’s unrounded panel, smoke is exactly 0 on every county-day with hms_share = 0, in all 20 years. After rounding, smoke_pm25 = 0.00 also occurs on 354 to 1,219 plume county-days per year, and 7 county-days in 2006–2025 show hms_share = 0.000 with smoke_pm25 = 0.01 (06073 on 2007-11-01, 53033 on 2012-09-21, 06085 on 2013-07-30, 12055 on 2015-03-20, 37173 on 2019-05-29, 32003 on 2020-09-26, 37173 on 2021-04-06). A check of the zero rule on the rounded files would fail in 2007, 2012, 2013, 2015, 2019, 2020, and 2021; the student script runs it on unrounded values.
• Background can be slightly negative. In July–September 2016, 11 counties in North Dakota and Minnesota (1,012 county-days, minimum −0.69 µg/m³), and in July–September 2022, Douglas County, Wisconsin (92 county-days, −0.35), have a negative background. Total PM2.5 is negative on 217 to 2,478 county-days per year.
• Monthly panel. n_days_obs equals the days in the month in every row and complete_month is always TRUE: the project’s daily panel has no gaps. Monthly smoke averages 0.597 µg/m³ (minimum −1.94, maximum 161.77), and 7.2% of county-months are negative.
• Releases. 2006–2023 come from one CSU release (2025-03-21), and 2024 and 2025 from two later ones (2026-02-25 and 2026-08-06). csu_release is the Dryad publication date; the NetCDF creation dates are 2025-01-13, 2025-10-22, and 2026-07-30. The “Downloaded” dates in the project’s README_sources.md are file timestamps kept from the provider’s archive (they match the NetCDF creation dates to within a day), not download dates; all files were downloaded from Dryad version 8. If CSU revises a year, re-run the project build and then this export.
• Arrow threads. The script reads Parquet with options(arrow.use_threads = FALSE), as the project build does: the project’s build log recorded garbled rows from a multi-threaded read with arrow 24.0.0.
• Memory and time. The export runs in about 3 to 6 minutes, depending on how busy the machine is. Its peak memory was 1.6 to 2.0 GB in our runs; in an earlier version without the gc() call in export_csu_year(), R held on to each year’s tables and the peak passed 2.2 GB.

ERA5 winds and surface weather: instructor export

The raw ERA5 files take 17.1 GB and can only be downloaded with a Copernicus account, so the bundle ships none of them. This export gives students the county results for every year instead. It reads the research project’s county-day tables, which the project’s script 02_winds.R built from the raw bundles B2 (500 hPa geopotential) and B3 (surface fields) as population-weighted county means (see the ERA5 section). It keeps ten weather columns, rounds them, joins the surface fields to the 500 hPa height, and writes one CSV file per year.

Students do not run this script: it needs the research project folder, which is not in the bundle. They use its output files, which are documented in the ERA5 section and below. The script is R/m09x_instructor_export.R. Run it from the bundle root with the project folder next to the bundle (../wildfire-claude), or set the environment variable WILDFIRE_PROJECT_DIR to another copy. Its setup chunk loads tidyverse and arrow. The validation script R/check_m09.R compares the export with the project files. The project has no county table of the 850/700 hPa winds (bundle B1); it keeps them only as gridded NetCDF files on the local solar day (data/cleaned/winds/winds_layer/winds_layer_<year>.nc), so the export has no winds above the surface.

Provider	Research project build (scripts/R/02_winds.R) from ERA5 daily means (ECMWF for the Copernicus Climate Change Service), bundles B2 and B3
Coverage	Project tables: 2003-01-01 to 2026-03-31 (surface) and 1991-01-01 to 2026-03-31 (500 hPa); export: 2006–2025, daily · the 3,108 contiguous-U.S. counties (DC included)
Raw unit	one Parquet row = a county × UTC day
Clean unit	one row = a county × UTC day (one file per year)
Licence and citation	CC BY 4.0; cite Hersbach et al. (2020) and the ERA5 datasets as in the ERA5 section
Documentation	ERA5 section of this guide; project codebook data/cleaned/winds/codebook_winds.csv; project script scripts/R/02_winds.R (header and sections 2–3)

Raw files

These are the inputs, in the research project folder (default ../wildfire-claude, next to the bundle). They are not in the bundle.

File (in the project’s data/cleaned/winds/)	Size	Format	One row is
sl_county_day/era5_sl_county_day_<year>.parquet, 24 files for 2003–2026 (the export reads 2006–2025)	86.2–86.5 MB each (2026, January–March only: 22.7 MB); 1,726.2 MB for 2006–2025	Apache Parquet (a compressed, column-based table format), 2 row groups per file	a county × UTC day: 1,134,420 rows per year (1,137,528 in leap years); 26,390,028 rows for 2003-01-01 to 2026-03-31
z500_county_day.parquet	749.9 MB	Apache Parquet, 39 row groups (blocks of about one million rows), sorted by county and then date	a county × UTC day, 1991-01-01 to 2026-03-31: 40,012,392 rows (3,108 counties × 12,874 days)
codebook_winds.csv	0.003 MB	CSV	one variable of the two tables: label, units, source, construction, and summary statistics

Raw variables

The yearly files sl_county_day/era5_sl_county_day_<year>.parquet have these columns. Every weather value is a population-weighted county mean of daily (UTC-day) means of hourly ERA5 values; the weights come from 2020 block-group centers of population.

Variable	Type	Description
GEOID	string	5-character county FIPS code, 2020 boundaries (exported)
date	date32	UTC day (exported)
ssrd_wm2	double	Surface solar radiation downwards, W/m²: the daily mean of hourly accumulations (J/m²) divided by 3,600 (exported)
blh_m	double	Boundary-layer height, m (exported)
t2m_c	double	2 m temperature, °C (t2m − 273.15) (exported)
d2m_c	double	2 m dewpoint temperature, °C (d2m − 273.15) (exported)
vp_hpa	double	Vapor pressure, hPa: the Magnus saturation vapor pressure at the dewpoint, computed per grid cell, then averaged (not exported: it carries the same information as d2m_c)
rh_pct	double	Relative humidity, %: 100 × e(d2m) / e(t2m) per grid cell from the daily-mean temperature and dewpoint, capped at 100, then averaged (exported)
u10_ms	double	10 m eastward wind, m/s (exported)
v10_ms	double	10 m northward wind, m/s (exported)
ws10_ms	double	Speed of the county-mean daily wind, √(u10_ms² + v10_ms²), m/s (exported)
source_product	string	“ERA5 post-processed daily statistics (Copernicus CDS), daily_mean, UTC day” in every row, including the months the project computed from hourly data with the same rule (not exported)
download_date	string	Latest download date among the year’s raw files, e.g., 2026-10-01 for 2006 (not exported)
weight_basis	string	“population (Census 2020 block-group centres of population)” in every row (not exported)

z500_county_day.parquet has these columns.

Variable	Type	Description
GEOID	string	5-character county FIPS code (exported)
date	date32	UTC day (exported)
z500_gpm	double	500 hPa geopotential height, gpm: geopotential / 9.80665 (exported)
z500_anom_gpm	double	z500_gpm minus the county’s 1991–2005 day-of-year climatology (a constant plus sine and cosine waves with periods of 1 year, 6 months, and 4 months, fitted county by county; no trend removed) (exported)
z500_bin	int32	Anomaly quintile bin, 1 (lowest) to 5, from county-specific cut points of 1991–2005 May–September days, applied to every day (not exported: it belongs to the project’s design)
source_product	large_string	As in the surface files (not exported)
download_date	string	Latest download date among all B2 raw files: 2026-10-04 in every row (not exported)
weight_basis	large_string	As in the surface files (not exported)

Cleaning steps

The script R/m09x_instructor_export.R makes these decisions:

1. Check coverage first. Both tables must cover every day of 2006–2025. They do: the surface table runs from 2003-01-01 to 2026-03-31 (26,390,028 rows) and the 500 hPa table from 1991-01-01 to 2026-03-31 (40,012,392 rows), both for all 3,108 counties.
2. Ten weather columns. It keeps temperature, dewpoint, humidity, the 10 m wind (u, v, and speed), boundary-layer height, sunlight, and the 500 hPa height and anomaly. It drops vp_hpa (same information as the dewpoint), z500_bin (a project design choice), and the three provenance columns, which hold the same text in every row of a file.
3. Read the 500 hPa file in blocks. The file is sorted by county and then date, so every row group holds days from every year. The script reads one row group at a time and keeps only the days of the year it is exporting, which keeps memory low.
4. Rounding. Temperatures and winds to 2 decimals; humidity, sunlight, and the 500 hPa height and anomaly to 1; boundary-layer height to whole meters.
5. Checks before writing. Each year must have every county on every day, unique keys, no missing values, and values in physical ranges (temperature between −50 and 50 °C, dewpoint not above temperature, humidity above 0 and at most 100%, non-negative speed and sunlight, positive boundary-layer height, and 500 hPa height between 4,500 and 6,500 gpm).

The first chunk sets the paths and the years to export.

# INSTRUCTOR ONLY. The raw ERA5 files are 17 GB and need a Copernicus account,
# so students get the research project's county-day weather tables instead.
# The project folder sits next to the bundle; set WILDFIRE_PROJECT_DIR to use
# another copy.
project_dir <- Sys.getenv("WILDFIRE_PROJECT_DIR", unset = "../wildfire-claude")
winds_dir <- file.path(project_dir, "data", "cleaned", "winds")
sl_dir <- file.path(winds_dir, "sl_county_day")
z500_file <- file.path(winds_dir, "z500_county_day.parquet")
stopifnot(dir.exists(sl_dir), file.exists(z500_file))

# Read Parquet single-threaded, as the project build does: its build log
# recorded garbled rows from a threaded arrow read (arrow 24.0.0, 2026-09-29).
options(arrow.use_threads = FALSE)

era5_years <- 2006:2025
day_dir <- file.path(clean_dir, "era5_county_day")
dir.create(day_dir, recursive = TRUE, showWarnings = FALSE)

Check the date range and county count of both tables before exporting.

# Check year coverage before exporting. The project keeps surface weather in
# one Parquet file per year (2003 to March 2026) and the 500 hPa height in one
# file (1991 to March 2026). The summaries run inside arrow, so no table of
# county-days reaches R.
sl_files <- file.path(sl_dir, str_c("era5_sl_county_day_", 2003:2026, ".parquet"))
stopifnot(all(file.exists(sl_files)))

sl_coverage <- open_dataset(sl_files) |>
  summarize(first_day = min(date), last_day = max(date), rows = n(),
            counties = n_distinct(GEOID)) |>
  collect()
z500_coverage <- open_dataset(z500_file) |>
  summarize(first_day = min(date), last_day = max(date), rows = n(),
            counties = n_distinct(GEOID)) |>
  collect()
bind_rows(surface = sl_coverage, z500 = z500_coverage, .id = "table")

# Both tables must cover every day of the export years.
stopifnot(
  sl_coverage$first_day <= as.Date(str_c(min(era5_years), "-01-01")),
  sl_coverage$last_day >= as.Date(str_c(max(era5_years), "-12-31")),
  z500_coverage$first_day <= as.Date(str_c(min(era5_years), "-01-01")),
  z500_coverage$last_day >= as.Date(str_c(max(era5_years), "-12-31"))
)

The county frame from module m00 defines the complete set of counties.

# The county frame written by module m00: 3,108 contiguous counties plus DC.
counties <- read_csv(file.path(clean_dir, "counties_2020.csv"),
                     col_types = cols_only(GEOID = col_character()))
stopifnot(nrow(counties) == 3108, !anyDuplicated(counties$GEOID))

A reader for the large 500 hPa file that returns the rows of a date range.

# The z500 file holds 40 million county-days, sorted by county and then date,
# so every part of the file holds every year. A filtered arrow scan of it
# held about 1 GB at once. Reading it one row group (a block of about one
# million rows) at a time, and keeping only the days we need, stays near
# 0.6 GB.
z500_reader <- ParquetFileReader$create(z500_file)
z500_cols <- match(c("GEOID", "date", "z500_gpm", "z500_anom_gpm"),
                   names(z500_reader$GetSchema())) - 1L   # arrow counts from 0
stopifnot(!anyNA(z500_cols))

read_z500_days <- function(first_day, last_day) {
  map(seq_len(z500_reader$num_row_groups) - 1L, \(i) {
    as_tibble(z500_reader$ReadRowGroup(i, z500_cols)) |>
      filter(date >= first_day, date <= last_day)
  }) |>
    list_rbind()
}

This function exports one year: it reads the year’s surface file and the year’s 500 hPa rows, joins and rounds them, checks the grain, and writes the CSV.

# Export one year of county-day weather: the surface fields from that year's
# Parquet file, plus the 500 hPa height for the same days.
export_era5_year <- function(year) {
  first_day <- as.Date(str_c(year, "-01-01"))
  last_day <- as.Date(str_c(year, "-12-31"))
  year_days <- seq(first_day, last_day, by = "day")

  surface <- read_parquet(
    file.path(sl_dir, str_c("era5_sl_county_day_", year, ".parquet")),
    col_select = c(GEOID, date, t2m_c, d2m_c, rh_pct, u10_ms, v10_ms, ws10_ms,
                   blh_m, ssrd_wm2)
  )
  z500 <- read_z500_days(first_day, last_day)

  # Round to the precision the data can support; the CSV stays small.
  out <- surface |>
    inner_join(z500, by = c("GEOID", "date")) |>
    transmute(
      GEOID,
      date = as.Date(date),
      t2m_c = round(t2m_c, 2),
      d2m_c = round(d2m_c, 2),
      rh_pct = round(rh_pct, 1),
      u10_ms = round(u10_ms, 2),
      v10_ms = round(v10_ms, 2),
      ws10_ms = round(ws10_ms, 2),
      blh_m = round(blh_m),
      ssrd_wm2 = round(ssrd_wm2, 1),
      z500_gpm = round(z500_gpm, 1),
      z500_anom_gpm = round(z500_anom_gpm, 1)
    ) |>
    arrange(GEOID, date)

  # Grain: one row per county x day of the year, complete, no missing values,
  # and every value in a physically possible range.
  stopifnot(
    nrow(surface) == nrow(counties) * length(year_days),
    nrow(z500) == nrow(counties) * length(year_days),
    nrow(out) == nrow(counties) * length(year_days),
    nrow(distinct(out, GEOID, date)) == nrow(out),
    setequal(out$GEOID, counties$GEOID),
    setequal(out$date, year_days),
    !anyNA(out),
    all(between(out$t2m_c, -50, 50)),
    all(out$d2m_c <= out$t2m_c + 0.01),
    all(out$rh_pct > 0 & out$rh_pct <= 100),
    all(out$ws10_ms >= 0),
    all(out$blh_m > 0),
    all(out$ssrd_wm2 >= 0),
    all(between(out$z500_gpm, 4500, 6500))
  )

  out_file <- file.path(day_dir, str_c("era5_county_day_", year, ".csv.gz"))
  write_csv(out, out_file, na = "")

  # One summary row per year for the log.
  year_summary <- tibble(
    year = year, rows = nrow(out), mean_t2m_c = round(mean(out$t2m_c), 2),
    mean_ws10_ms = round(mean(out$ws10_ms), 2),
    mean_z500_anom_gpm = round(mean(out$z500_anom_gpm), 1),
    size_mb = round(file.size(out_file) / 1e6, 1)
  )
  # Free this year's tables before the next one, so peak memory stays near
  # one year's worth.
  rm(surface, z500, out)
  invisible(gc())
  year_summary
}

Run it for all 20 years and print a summary row per year.

# One year at a time. The years can also be run in pieces, e.g.
# map(2006:2010, export_era5_year), if a session is short on time.
day_summary <- map(era5_years, export_era5_year) |> list_rbind()
stopifnot(nrow(day_summary) == length(era5_years))
print(day_summary, n = Inf)

Clean files

File (in wildfire_data/clean/)	Rows	One row is	Size
era5_county_day/era5_county_day_<year>.csv.gz, 20 files for 2006–2025	1,134,420 per year (1,137,528 in 2008, 2012, 2016, 2020, 2024); 22,703,940 in all	a county × UTC day	28.5–28.7 MB each (about 80 MB uncompressed); 571.5 MB in all

The summary the script prints, one row per year: rows, means over all county-days, and file size.

Year	Rows	Mean t2m_c (°C)	Mean ws10_ms (m/s)	Mean z500_anom_gpm (gpm)	Size
2006	1,134,420	13.86	2.77	4.1	28.6 MB
2007	1,134,420	13.60	2.72	11.9	28.6 MB
2008	1,137,528	12.74	2.82	−9.8	28.7 MB
2009	1,134,420	12.69	2.77	−5.6	28.6 MB
2010	1,134,420	13.21	2.78	−7.6	28.6 MB
2011	1,134,420	13.51	2.81	−0.3	28.6 MB
2012	1,137,528	14.56	2.71	10.8	28.6 MB
2013	1,134,420	12.67	2.76	1.0	28.6 MB
2014	1,134,420	12.38	2.80	−6.9	28.6 MB
2015	1,134,420	13.79	2.68	22.0	28.5 MB
2016	1,137,528	14.26	2.73	18.4	28.6 MB
2017	1,134,420	14.08	2.78	15.0	28.6 MB
2018	1,134,420	13.43	2.70	18.6	28.5 MB
2019	1,134,420	13.35	2.69	13.2	28.5 MB
2020	1,137,528	13.95	2.74	19.1	28.6 MB
2021	1,134,420	14.03	2.70	17.7	28.5 MB
2022	1,134,420	13.48	2.83	11.6	28.6 MB
2023	1,134,420	14.30	2.67	11.4	28.5 MB
2024	1,137,528	14.39	2.73	22.2	28.6 MB
2025	1,134,420	13.83	2.70	17.8	28.6 MB

Clean variables

The 20 yearly files share these columns (the same as in the ERA5 section). Rows are sorted by GEOID and date, and no cell is missing.

Variable	Type	Units	Description
GEOID	chr	—	5-character county FIPS code, 2020 boundaries; joins to counties_2020.csv
date	date	—	UTC day
t2m_c	dbl	°C	2 m air temperature (project t2m_c). 2 decimals
d2m_c	dbl	°C	2 m dewpoint temperature (project d2m_c). 2 decimals
rh_pct	dbl	%	Relative humidity (project rh_pct). 1 decimal
u10_ms	dbl	m/s	10 m eastward wind (project u10_ms). 2 decimals
v10_ms	dbl	m/s	10 m northward wind (project v10_ms). 2 decimals
ws10_ms	dbl	m/s	Speed of the daily-mean 10 m wind (project ws10_ms, computed before rounding). 2 decimals
blh_m	dbl	m	Boundary-layer height (project blh_m). Whole meters
ssrd_wm2	dbl	W/m²	Surface solar radiation, 24-hour mean (project ssrd_wm2). 1 decimal
z500_gpm	dbl	gpm	500 hPa geopotential height (project z500_gpm). 1 decimal
z500_anom_gpm	dbl	gpm	500 hPa height anomaly against the 1991–2005 seasonal normal (project z500_anom_gpm). 1 decimal

Checks and cautions

• Exact copy of the project tables, up to rounding. R/check_m09.R re-reads every exported file and joins it to the project’s Parquet tables. All 20 files have the expected rows (3,108 counties × every day of the year), every row matches a project row, no GEOID–date pair is duplicated, and no value is missing. Every exported value equals the project’s value rounded to the stated decimals.
• Coverage beyond 2006–2025. The project tables also hold 2003–2005 (surface) and 1991–2005 (500 hPa), and both run to March 31, 2026. To export other years, change era5_years in the chunk m09x-paths; a 2026 file would be partial, and the grain check would stop it as written.
• No winds above the surface. The 850 and 700 hPa winds exist only as gridded fields, averaged over the two levels and shifted to each grid cell’s local solar day, with February 29 folded into February 28. Making county tables of them would need a new aggregation step; this export does not do it.
• The anomaly baseline is 1991–2005. No trend is removed, so later years run high: the yearly mean anomaly ranges from −9.8 gpm (2008) to +22.2 gpm (2024), and every year since 2015 is above +11 gpm.
• Arrow threads. The script reads Parquet with options(arrow.use_threads = FALSE), as the project build does: the project’s build log recorded garbled rows from a multi-threaded read with arrow 24.0.0.
• Time and memory. The export took 3.6 minutes in our final run (5.5 and 12.8 minutes in earlier runs on a busier machine). Its peak memory footprint was 1.8 GB. R/m09_era5.R peaked at 1.7 GB and R/check_m09.R at 2.1 GB.
• Plots in batch runs. R/m09_era5.R draws two plots. In RStudio or Quarto they appear on screen, but a plain Rscript R/m09_era5.R writes them to Rplots.pdf in the project root. For batch validation runs, use Rscript -e 'pdf(NULL); source("R/m09_era5.R", print.eval = TRUE)', and before uploading the bundle delete any Rplots.pdf from its root, together with the ._* files.
• Rebuilds. If the project rebuilds its wind tables (for example, after new ERA5 downloads), re-run this export and then R/m09_era5.R.

Smoke episodes and Canadian fire shares: instructor export

The episode tables and county-day fire shares are research-project products. Rebuilding them needs all 20 years of CSU smoke grids (20 GB), the project’s full MODIS fire record, and SEER populations, so students get the project’s finished tables instead. This export reads the files that the project script scripts/R/06_episode_count.R wrote to data/cleaned/episodes/, gives the columns consistent names, rounds them, fills in the missing smoke-bin rows, attaches each county-day’s smoke value from the project’s CSU panel, and writes four files to wildfire_data/clean/.

Students do not run this script: it needs the research project folder, which is not in the bundle. The script is R/m11x_instructor_export.R. Run it from the bundle root with the project folder next to the bundle (../wildfire-claude), or set the environment variable WILDFIRE_PROJECT_DIR to another copy. Its setup chunk loads tidyverse and arrow; it is not shown here. Every chunk below carries #| eval: false, so rendering the guide never runs the export or overwrites the clean files. The validation script R/check_m11.R compares the exports with the project files and rebuilds the episodes from the definitions.

Provider	Research project build, scripts/R/06_episode_count.R (27 September 2026), from CSU kriged smoke PM2.5 (Dryad, version 8), NASA FIRMS MODIS fire detections, and NCI SEER county populations
Coverage	2006–2025, daily · the 3,108 contiguous-U.S. counties (DC included), 9 regions
Raw unit	one row = a region × episode (episode_list.csv); a region × year (canadian_dominated_summary.csv); a region × year × smoke bin with person-days (person_days_by_bin.csv); a county × day with positive smoke (county_day_fire_share.parquet); a weighting × county × day (CSU panel)
Clean unit	one row = a region × episode; a region × year; a region × year × smoke bin (complete); a county × day with positive smoke
Licence and citation	Derived tables; cite the CSU, FIRMS, and SEER data as in the main section of this module
Documentation	Main section of this module; project codebook data/cleaned/codebook.csv; project build summary quality_reports/data_build/episode_count.md

Raw files

These are the inputs, in the research project folder (default ../wildfire-claude, next to the bundle). They are not in the bundle.

File (in the project’s data/cleaned/)	Size	Format	One row is
episodes/episode_list.csv	0.08 MB	comma-separated text, header row; text with commas in double quotes	a region × smoke episode (512 rows)
episodes/canadian_dominated_summary.csv	0.02 MB	comma-separated text, header row	a region × year (180 rows)
episodes/person_days_by_bin.csv	0.14 MB	comma-separated text, header row; bin labels with commas in double quotes	a region × year × smoke bin that has person-days (893 rows; empty bins are left out)
episodes/county_day_fire_share.parquet	80.4 MB	Apache Parquet (a compressed column-based table format)	a county × day with positive CSU smoke PM2.5 (2,876,135 rows)
panel/csu_county_day/background=median/year=<year>/part-0.parquet, 20 files for 2006–2025	23.8–32.1 MB each; 528.6 MB in all	Apache Parquet in a “hive” folder layout: the folder names hold the partition keys background and year	a weighting × county × day; only the smoke column is used here (documented in the CSU appendix)

The episodes folder also holds person_days_by_bin.parquet (the same table as the CSV), hms_person_days.csv (person-days under HMS medium and heavy smoke plumes, by region and year), and episode_count.rds (an R list with the same tables and the build settings). They are not exported.

Raw variables

episodes/episode_list.csv. The project’s codebook does not list this file; the descriptions come from the build script.

Variable	Type	Description
region_type	text	Census region or Eastern split
region	text	Region name: four Census regions; four eastern groups and their union East (4 splits)
start	text date (YYYY-MM-DD)	First day of the episode
end	text date (YYYY-MM-DD)	Last day of the episode
n_days	integer	Days in the episode (at least 2)
peak_pw_smoke	number	Highest daily population-weighted smoke PM2.5 of the region in the episode, µg/m³
mean_share_ca	number	Smoke-weighted mean over the episode’s days of the region’s daily population-weighted Canadian share of same-day FRP within 1,500 km
person_days_gt15	number	Sum over the episode’s days of the population of counties with smoke PM2.5 above 15 µg/m³
canadian_dominated	TRUE/FALSE	mean_share_ca ≥ 0.5
conc_source	text in double quotes	Concentration source; in this build always CSU kriged smoke PM2.5 (median background, population-weighted)

episodes/canadian_dominated_summary.csv (also not in the codebook; descriptions from the build script):

Variable	Type	Description
region_type, region	text	As in the episode list
year	integer	Calendar year
pop	integer	Region population (SEER, sum over counties; 2024 values for 2025)
pd_gt15	integer	Person-days with county smoke PM2.5 above 15 µg/m³
pd_gt15_ca	integer	Of these, person-days on county-days with same-day Canadian share ≥ 0.5
pd_gt15_ca_lag03	integer	The same with the d−3 to d share
pd_gt35	integer	Person-days above 35 µg/m³
pd_gt35_ca	integer	Of these, Canadian-dominated (same day)
days_gt15_ca_per_capita	number	pd_gt15_ca ÷ pop
ca_share_of_gt15	number	pd_gt15_ca ÷ pd_gt15; empty when pd_gt15 is 0
pop_share_any_gt15_ca	number	Share of residents living in a county with at least one Canadian-dominated day above 15 µg/m³ that year

episodes/person_days_by_bin.csv (labels quoted from the project codebook, which documents the Parquet copy):

Variable	Type	Description
region_type	text	“Census region or eastern split”
region	text	“Census region or eastern-split name”
year	integer	“Calendar year of the smoke day”
bin	text	“Daily smoke PM2.5 bin (ug/m3)”: 0 / no smoke, (0,5], (5,15], (15,35], (35,55], >55
person_days	number	“Person-days (SEER population x days)”
person_days_ca_dom	number	“Person-days with share_ca_sameday >= 0.5”; empty in the no-smoke bin
person_days_ca_dom_lag03	number	“Person-days with share_ca_lag03 >= 0.5”; empty in the no-smoke bin
person_days_no_fire	number	“Person-days with no FRP within 1,500 km”; empty in the no-smoke bin
pop	integer	“Region population”
conc_source	text in double quotes	“Concentration product used” (CSU in this build)
person_days_per_capita	number	“Person-days / population”

episodes/county_day_fire_share.parquet (labels quoted from the project codebook):

Variable	Type	Description
GEOID	string	“County FIPS (TIGER 2020, CT legacy counties)”
date	date32	“Calendar date”
frp_CA_sameday	double	“Canadian FRP within 1,500 km, same local day (MW)”
frp_US_sameday	double	“US FRP within 1,500 km, same day” (MW)
frp_MX_sameday	double	“Mexican FRP within 1,500 km, same day” (MW)
frp_CA_lag03	double	“Canadian FRP within 1,500 km, days d-3..d” (MW)
frp_US_lag03	double	“US FRP, days d-3..d” (MW)
frp_MX_lag03	double	“Mexican FRP, days d-3..d” (MW)
share_ca_sameday	double	“Canadian share of same-day FRP”; missing when all three same-day sums are 0
share_ca_lag03	double	“Canadian share of d-3..d FRP”

From the CSU panel the export uses weight_basis (keeps population), GEOID, date, and smoke (smoke PM2.5 in µg/m³, negatives kept; the export keeps rows with smoke > 0). The CSU appendix describes all of its columns.

How the project computed the FRP sums: it took MODIS fire detections (Terra and Aqua) from one FIRMS archive delivery, kept vegetation fires (type 0) with confidence of at least 30, assigned each detection to Canada (StatCan 2021 provinces), the U.S. (TIGER 2020 states, Alaska included, Hawaii excluded), or Mexico by its location, and summed FRP by 1° cell and local solar day. On days when one satellite was out, it filled that satellite’s share from the other satellite (its column frp_interp_v2). For each county it then added the cells whose center lies within 1,500 km of the Census internal point, using great-circle (haversine) distance on a sphere of radius 6,371 km.

Cleaning steps

The script R/m11x_instructor_export.R makes these decisions:

1. CSU build only. The project script picks its smoke source when it runs: CSU when the CSU record is complete, otherwise a Childs et al. stand-in. Everything in the guide describes the CSU build, so the export stops if any conc_source value is not the CSU text.
2. Consistent names, same meaning. Episode columns become start_date, end_date, peak_smoke_pm25, and share_ca_mean, with an episode_id and the start year added. In the summary, pd_ becomes person_days_ and ca_share_of_gt15 becomes share_gt15_ca. In the bins, bin becomes smoke_bin, person_days_ca_dom becomes person_days_ca, and person_days_per_capita becomes days_per_capita. FRP columns use lower-case country codes. The constant conc_source column is dropped.
3. Rounding. PM2.5 to 2 decimals, FRP to 1, county-day shares to 4, episode and summary shares and per-capita days to 3. Person-days and populations are whole numbers and stay exact.
4. Complete bins. The project leaves out a bin when no county-day of the region-year fell in it. The export adds those 187 rows with zero person-days (complete() with explicit = FALSE, so the no-smoke bin keeps its empty fire columns).
5. Smoke attached, keys checked. The fire-share rows are exactly the county-days with positive CSU smoke. The export joins the smoke value from the CSU panel and stops unless every row finds its value and the row counts match.
6. Flags before rounding. ca_dominated_sameday and ca_dominated_lag03 are computed from the unrounded shares, so they reproduce the project’s definition exactly; 27 same-day shares (29 lag shares) would cross 0.5 after rounding to 4 decimals.
7. One year at a time. The fire-share file is built one year at a time, so at most about 0.45 million county-days (2023, the largest year) are in R at once. Each year is appended to one open gzip connection, with the column names written for the first year only; the full 2.9-million-row table never sits in memory.

The first chunk sets the paths and the expected concentration source.

# INSTRUCTOR ONLY. The smoke episodes and fire shares are research-project
# products (project script scripts/R/06_episode_count.R). Rebuilding them needs
# 20 GB of CSU grids and the full MODIS fire record, so students get the
# project's tables instead. The project folder sits next to the bundle; set
# WILDFIRE_PROJECT_DIR to use another copy.
project_dir <- Sys.getenv("WILDFIRE_PROJECT_DIR", unset = "../wildfire-claude")
episodes_dir <- file.path(project_dir, "data", "cleaned", "episodes")
csu_dir <- file.path(project_dir, "data", "cleaned", "panel", "csu_county_day",
                     "background=median")
stopifnot(dir.exists(episodes_dir), dir.exists(csu_dir))

# Read Parquet single-threaded, as the project build does: its build log
# recorded garbled rows from a threaded arrow read (arrow 24.0.0, 2026-09-29).
options(arrow.use_threads = FALSE)

# The project build picks its concentration source at run time: CSU smoke when
# the CSU record is complete, otherwise a Childs stand-in. Everything below
# (and the guide) describes the CSU build, so the export stops on any other.
csu_source <- "CSU kriged smoke PM2.5 (median background, population-weighted)"
export_years <- 2006:2025

The county frame and the nine regions are used to check that every region-year is present.

# The county frame written by module m00 and the nine regions of the project
# tables: four Census regions, four eastern groups, and their union.
counties <- read_csv(file.path(clean_dir, "counties_2020.csv"),
                     col_types = cols_only(GEOID = col_character()))
stopifnot(nrow(counties) == 3108, !anyDuplicated(counties$GEOID))

regions <- tribble(
  ~region_type,    ~region,
  "Census region", "Midwest",
  "Census region", "Northeast",
  "Census region", "South",
  "Census region", "West",
  "Eastern split", "East (4 splits)",
  "Eastern split", "Great Lakes",
  "Eastern split", "Mid-Atlantic",
  "Eastern split", "New England",
  "Eastern split", "Upper Midwest"
)
region_years <- regions |> cross_join(tibble(year = export_years))

The episode list gets an id, a year, clearer names, and checks: runs of consecutive days, no overlaps within a region, and a rounded share that still gives the project’s flag.

# Episode list: one row per run of 2 or more consecutive days on which a
# region's population-weighted smoke PM2.5 was at least 5 ug/m3. We add an id
# and the start year, give the columns clearer names, and drop the constant
# conc_source column after checking it.
episodes_raw <- read_csv(
  file.path(episodes_dir, "episode_list.csv"),
  col_types = cols(
    region_type = col_character(), region = col_character(),
    start = col_date(), end = col_date(), n_days = col_integer(),
    peak_pw_smoke = col_double(), mean_share_ca = col_double(),
    person_days_gt15 = col_double(), canadian_dominated = col_logical(),
    conc_source = col_character()
  )
)
stopifnot(all(episodes_raw$conc_source == csu_source))

episodes <- episodes_raw |>
  arrange(region_type, region, start) |>
  transmute(
    episode_id = row_number(),
    region_type, region,
    start_date = start,
    end_date = end,
    year = as.integer(year(start)),
    n_days,
    peak_smoke_pm25 = round(peak_pw_smoke, 2),
    share_ca_mean = round(mean_share_ca, 3),
    person_days_gt15,
    canadian_dominated
  )

# Grain: one row per region x episode. Episodes are runs of consecutive days,
# so they never overlap within a region. The project's run detection ignores
# year boundaries, so an episode could run from December into January. None
# does in this build; the year check below stops the export if a rebuild adds
# one, because the guide says no episode crosses into a new year.
stopifnot(
  nrow(episodes) == nrow(episodes_raw),
  !anyDuplicated(select(episodes, region, start_date)),
  all(paste(episodes$region_type, episodes$region) %in%
        paste(regions$region_type, regions$region)),
  all(episodes$n_days >= 2),
  all(as.integer(episodes$end_date - episodes$start_date) + 1L == episodes$n_days),
  all(year(episodes$end_date) == episodes$year),
  all(episodes$year %in% export_years),
  all(episodes$peak_smoke_pm25 >= 5),
  all(between(episodes$share_ca_mean, 0, 1)),
  all(episodes$person_days_gt15 >= 0),
  # The rounded share still gives the project's flag.
  all(episodes$canadian_dominated == (episodes$share_ca_mean >= 0.5))
)
no_overlap <- episodes |>
  arrange(region, start_date) |>
  mutate(gap = as.integer(start_date - lag(end_date)), .by = region)
stopifnot(all(no_overlap$gap > 1, na.rm = TRUE))

write_csv(episodes, file.path(clean_dir, "smoke_episodes_2006_2025.csv"), na = "")
count(episodes, region_type, region, canadian_dominated) |>
  pivot_wider(names_from = canadian_dominated, values_from = n, values_fill = 0,
              names_prefix = "ca_dom_")

The region-year summary is renamed and checked for completeness and internal consistency.

# Region x year summary of person-days above 15 and 35 ug/m3, with the part
# flagged Canadian-dominated. The pd_ prefix becomes person_days_ so the
# names match the episode list; the meaning of every column is unchanged.
summary_raw <- read_csv(
  file.path(episodes_dir, "canadian_dominated_summary.csv"),
  col_types = cols(region_type = col_character(), region = col_character(),
                   year = col_integer(), .default = col_double())
)

ca_summary <- summary_raw |>
  transmute(
    region_type, region, year, pop,
    person_days_gt15 = pd_gt15,
    person_days_gt15_ca = pd_gt15_ca,
    person_days_gt15_ca_lag03 = pd_gt15_ca_lag03,
    person_days_gt35 = pd_gt35,
    person_days_gt35_ca = pd_gt35_ca,
    days_gt15_ca_per_capita = round(days_gt15_ca_per_capita, 3),
    share_gt15_ca = round(ca_share_of_gt15, 3),
    pop_share_any_gt15_ca = round(pop_share_any_gt15_ca, 3)
  ) |>
  arrange(region_type, region, year)

# Grain: one row per region x year, complete (9 regions x 20 years). The
# share of person-days above 15 that were Canadian-dominated is missing only
# when the region had no person-days above 15 that year.
stopifnot(
  nrow(ca_summary) == nrow(region_years),
  nrow(semi_join(ca_summary, region_years, by = c("region_type", "region", "year"))) ==
    nrow(region_years),
  !anyNA(select(ca_summary, -share_gt15_ca)),
  all(is.na(ca_summary$share_gt15_ca) == (ca_summary$person_days_gt15 == 0)),
  all(ca_summary$person_days_gt15_ca <= ca_summary$person_days_gt15),
  all(ca_summary$person_days_gt15_ca_lag03 <= ca_summary$person_days_gt15),
  all(ca_summary$person_days_gt35 <= ca_summary$person_days_gt15),
  all(ca_summary$person_days_gt35_ca <= ca_summary$person_days_gt35),
  all(between(ca_summary$pop_share_any_gt15_ca, 0, 1))
)

write_csv(ca_summary, file.path(clean_dir, "canadian_dominated_summary.csv"), na = "")

The bin table is completed to six bins per region-year; the six bins must add up to every resident on every day of the year.

# Person-days by daily smoke PM2.5 bin. The project file leaves out a bin
# when no county-day of that region-year fell in it; we add those rows with
# zero person-days so every region-year has all six bins.
bin_levels <- c("0 / no smoke", "(0,5]", "(5,15]", "(15,35]", "(35,55]", ">55")

bins_raw <- read_csv(
  file.path(episodes_dir, "person_days_by_bin.csv"),
  col_types = cols(region_type = col_character(), region = col_character(),
                   year = col_integer(), bin = col_character(),
                   conc_source = col_character(), .default = col_double())
)
stopifnot(all(bins_raw$conc_source == csu_source), all(bins_raw$bin %in% bin_levels))

bins <- bins_raw |>
  mutate(smoke_bin = factor(bin, levels = bin_levels)) |>
  complete(
    nesting(region_type, region, year, pop), smoke_bin,
    fill = list(person_days = 0, person_days_ca_dom = 0, person_days_ca_dom_lag03 = 0,
                person_days_no_fire = 0, person_days_per_capita = 0),
    # Fill only the added rows: the no-smoke bin keeps its missing fire columns.
    explicit = FALSE
  ) |>
  arrange(region_type, region, year, smoke_bin) |>
  transmute(
    region_type, region, year,
    smoke_bin = as.character(smoke_bin),
    person_days,
    person_days_ca = person_days_ca_dom,
    person_days_ca_lag03 = person_days_ca_dom_lag03,
    person_days_no_fire,
    pop,
    days_per_capita = round(person_days_per_capita, 3)
  )

# Grain: one row per region x year x bin (9 x 20 x 6). The six bins of a
# region-year add up to every resident on every day of the year. The fire
# columns do not apply to the no-smoke bin, so they are missing there only.
year_totals <- bins |>
  summarize(total = sum(person_days), pop = first(pop), n_pop = n_distinct(pop),
            .by = c(region_type, region, year)) |>
  mutate(days = if_else(leap_year(year), 366, 365))
zero_bin <- bins$smoke_bin == "0 / no smoke"
stopifnot(
  nrow(bins) == nrow(region_years) * length(bin_levels),
  !anyDuplicated(select(bins, region_type, region, year, smoke_bin)),
  nrow(bins) - nrow(bins_raw) == sum(bins$person_days == 0 & !zero_bin),
  all(year_totals$n_pop == 1),
  all(year_totals$total == year_totals$pop * year_totals$days),
  all(is.na(bins$person_days_ca) == zero_bin),
  all(is.na(bins$person_days_ca_lag03) == zero_bin),
  all(is.na(bins$person_days_no_fire) == zero_bin),
  all(bins$person_days >= 0),
  all(bins$person_days_ca <= bins$person_days, na.rm = TRUE),
  all(bins$person_days_ca_lag03 <= bins$person_days, na.rm = TRUE),
  all(bins$person_days_no_fire <= bins$person_days, na.rm = TRUE)
)

write_csv(bins, file.path(clean_dir, "smoke_person_days_by_bin.csv"), na = "")
nrow(bins) - nrow(bins_raw)

The fire-share function exports one year: it reads that year’s rows from the project file, joins the CSU smoke values, adds the flags, rounds, and checks.

# County-day fire shares. The project file has one row per county-day with
# positive CSU smoke PM2.5 (population-weighted, median background); other
# county-days have no row. We attach that smoke value from the project's CSU
# panel, so the file says why each row exists, and we add the Canadian-
# dominated flags.
share_file <- file.path(episodes_dir, "county_day_fire_share.parquet")

export_share_year <- function(year) {
  first_day <- as.Date(str_c(year, "-01-01"))
  last_day <- as.Date(str_c(year, "-12-31"))

  share <- open_dataset(share_file) |>
    filter(date >= first_day, date <= last_day) |>
    collect()
  smoke <- read_parquet(file.path(csu_dir, str_c("year=", year), "part-0.parquet"),
                        as_data_frame = FALSE) |>
    filter(weight_basis == "population", smoke > 0) |>
    select(GEOID, date, smoke) |>
    collect()

  out <- share |>
    left_join(smoke, by = c("GEOID", "date")) |>
    # Flags first, from the unrounded shares (a share of 0.49996 must stay
    # below 0.5 even though it rounds to 0.5000).
    mutate(
      ca_dominated_sameday = !is.na(share_ca_sameday) & share_ca_sameday >= 0.5,
      ca_dominated_lag03 = !is.na(share_ca_lag03) & share_ca_lag03 >= 0.5
    ) |>
    transmute(
      GEOID,
      date = as.Date(date),
      smoke_pm25 = round(smoke, 2),
      frp_ca_sameday = round(frp_CA_sameday, 1),
      frp_us_sameday = round(frp_US_sameday, 1),
      frp_mx_sameday = round(frp_MX_sameday, 1),
      frp_ca_lag03 = round(frp_CA_lag03, 1),
      frp_us_lag03 = round(frp_US_lag03, 1),
      frp_mx_lag03 = round(frp_MX_lag03, 1),
      share_ca_sameday = round(share_ca_sameday, 4),
      share_ca_lag03 = round(share_ca_lag03, 4),
      ca_dominated_sameday,
      ca_dominated_lag03
    ) |>
    arrange(GEOID, date)

  # Grain: one row per county-day with positive smoke; the rows are exactly
  # the CSU smoke-positive county-days of the year. The share is missing only
  # when no fire burned within 1,500 km that day.
  total_sameday <- share$frp_CA_sameday + share$frp_US_sameday + share$frp_MX_sameday
  stopifnot(
    nrow(out) == nrow(smoke),
    !anyNA(out$smoke_pm25),
    nrow(distinct(out, GEOID, date)) == nrow(out),
    all(out$GEOID %in% counties$GEOID),
    all(between(out$date, first_day, last_day)),
    all(select(out, starts_with("frp_")) >= 0),
    all(between(out$share_ca_sameday, 0, 1), na.rm = TRUE),
    all(between(out$share_ca_lag03, 0, 1)),
    sum(is.na(out$share_ca_sameday)) == sum(total_sameday == 0),
    !anyNA(select(out, -share_ca_sameday))
  )

  year_summary <- tibble(
    year = year, rows = nrow(out), counties = n_distinct(out$GEOID),
    na_share_sameday = sum(is.na(out$share_ca_sameday)),
    ca_dominated_sameday = sum(out$ca_dominated_sameday)
  )
  list(data = out, summary = year_summary)
}

The last chunk opens one gzip connection, runs the function for each year, and appends that year’s rows to the file (the header goes in with 2006 only). It then checks that the yearly row counts add up to the project file’s total.

# One year at a time: at most 0.45 million county-days are in R at once.
# Each year goes straight into one open gzip connection (header with the
# first year only), so the 2.9 million rows never sit in memory together.
share_out <- file.path(clean_dir, "county_day_fire_share_2006_2025.csv.gz")
share_con <- gzfile(share_out, open = "wb")
share_summary <- map(export_years, function(year) {
  one_year <- export_share_year(year)
  write_csv(one_year$data, share_con, na = "", col_names = year == export_years[1])
  one_year$summary
}) |>
  list_rbind()
close(share_con)
print(share_summary, n = Inf)

# The whole file must equal the project's row count. Keys are unique within
# each year (checked in the function) and the years do not overlap, so they
# are unique in the whole file too.
project_rows <- open_dataset(share_file) |> count() |> collect() |> pull(n)
stopifnot(sum(share_summary$rows) == project_rows,
          n_distinct(share_summary$year) == length(export_years))
round(file.size(share_out) / 1e6, 1)

Clean files

The export writes the four files described in the main section of this module:

File (in wildfire_data/clean/)	Rows	One row is	Size
smoke_episodes_2006_2025.csv	512	a region × smoke episode	0.04 MB
canadian_dominated_summary.csv	180	a region × year	0.02 MB
smoke_person_days_by_bin.csv	1,080	a region × year × smoke bin	0.08 MB
county_day_fire_share_2006_2025.csv.gz	2,876,135	a county × day with positive smoke PM2.5	57.7 MB (214.6 MB unzipped)

Rows of the fire-share file by year (all from the export’s log):

Year	Rows	Counties with a row	Same-day share missing	Canadian-dominated (same day)
2006	42,533	3,108	8	3,290
2007	74,700	3,106	0	3,107
2008	55,157	3,103	2	3,256
2009	37,504	3,097	0	2,741
2010	55,629	3,105	3	8,797
2011	116,815	3,106	17	4,120
2012	134,365	3,108	0	8,061
2013	97,655	3,108	0	10,824
2014	62,920	3,108	29	5,801
2015	75,504	3,108	9	9,266
2016	64,323	3,108	3	5,134
2017	89,097	3,108	8	13,527
2018	110,700	3,108	0	11,661
2019	98,650	3,108	5	6,128
2020	109,104	3,108	18	1,584
2021	253,694	3,108	34	45,506
2022	333,718	3,108	42	8,978
2023	448,264	3,108	86	88,607
2024	354,848	3,108	100	28,442
2025	260,955	3,108	34	44,784
All	2,876,135	3,108	398	313,614

Clean variables

The main section of this module describes every column of the four files.

Checks and cautions

• Health screen. R/check_m11.R read the column names of all 7 files in the project’s episodes folder (101 columns, including the names inside episode_count.rds) and of the four exports, and matched them against words such as death, mortality, ICD, hospital, visit, and outcome. Nothing matched: the files hold smoke, fire power, population, and region data only.
• Exports equal the project files. Episodes: 512 of 512, identical dates, lengths, person-days, and flags; peaks within 0.005 µg/m³ and shares within 0.0005 (rounding). Summary: 180 of 180 region-years, person-days and populations identical. Bins: the 893 project rows are identical; the 187 added rows (11 in (15,35], 63 in (35,55], 113 in >55) are all zero. Fire shares: 2,876,135 county-days, key for key, equal to the CSU panel’s positive-smoke county-days in every year; FRP within 0.05 MW, shares within 0.00005, smoke within 0.005 µg/m³, and both flags equal to the project’s definition in every row.
• Rebuild from the definitions. From the project’s CSU panel, SEER file, and fire shares, and the regions in the bundle’s counties_2020.csv, the check rebuilds all 512 episodes with the same start and end dates (peaks within 4 × 10⁻¹³ µg/m³, shares within 6 × 10⁻¹⁶, person-days above 15 identical), all 713 smoky bin rows, and all 180 values of pop_share_any_gt15_ca. So the definitions written in the main section are the ones the project used.
• Rebuild from bundle files. With the rounded smoke_pm25 and share_ca_sameday of the export, the bundle’s SEER file (which gives the project’s region populations exactly in all 180 region-years), and counties_2020.csv, the check finds the same 512 episodes with the same dates and flags; person-days above 15 differ by up to 1.5 million in a few episodes because of rounding at 15.00.
• Pre-amendment FRP series. The project later made Aqua-only FRP (frp_aqua_v2) its headline fire column, but these tables use frp_interp_v2 (Terra plus Aqua, outage fill without the season-boundary cut), the series the build script names. If the project rebuilds the tables with another FRP column, re-run this export and update the guide text.
• Re-run order. If the project re-runs 06_episode_count.R (for example after a new CSU release), re-run this export and R/check_m11.R. The episode ids are row numbers and may change.
• Arrow threads. The script reads Parquet with options(arrow.use_threads = FALSE), as the project build does: the project’s build log recorded garbled rows from a multi-threaded read with arrow 24.0.0.
• Memory and time. The export runs in about 40 seconds with a peak resident memory of about 1.3 GB (macOS peak footprint 1.6 GB); the largest single year, 2023, sets the peak. The check script runs in about 75 seconds and peaks at about 1.4 GB resident.
• Fragment check. R/check_m11.R also compares every chunk in this appendix and in the main section with the matching ## ---- label ---- block of the scripts, byte for byte, and checks that each appendix chunk has #| eval: false. Re-run it after any script edit.