# Agent Guide: End Use Load Profiles for the U.S. Building Stock **Audience:** automated consumers — LLM agents, pipelines, and scripts with S3 read access and no human to ask. **Bucket:** `oedi-data-lake`, region `us-west-2`, public read, no requester-pays. **Root prefix:** `nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/` **Verified:** 2026-08-05. Every key in this guide was confirmed to exist. > **Read this before writing any key.** This repository contains three traps that fail > silently — returning no data and no error rather than an exception — in §2, §5.1 and > §5.2. All three have caused wrong answers. > > §7 is a fourth kind of failure and the one most likely to bite you: **13 known issues > affect published results**, and reading the affected columns returns numbers that look > entirely normal. Nothing in the data signals it. Check `known_issues` in the release > manifest before you trust any figure. --- ## 1. Decide what you need before you list anything `ListObjectsV2` against this repository is expensive and misleading. One directory holds 347,147 objects. Resolve your target from the tables in this guide, then fetch a single key. | You want | Directory | Granularity | |---|---|---| | Building characteristics + annual energy | `metadata_and_annual_results/` | One row per building per upgrade | | Load shapes for a geography | `timeseries_aggregates/` | 15-minute, summed over buildings | | Load shape for one building | `timeseries_individual_buildings/` | 15-minute, one building | | The simulation input model | `building_energy_models/` | One model per building per upgrade | | Weather driving the simulation | `weather/` | Hourly, per county | **Sample-size caveat.** A single row is not a household or a building; it is a weighted sample. The ResStock team recommends **at least 1,000 dwelling models** before drawing a conclusion, to hold uncertainty near 15%. County-level aggregates in sparsely populated counties may rest on single-digit model counts and will look implausible. Always read the `models_used` column before trusting an aggregate. --- ## 2. Pick a release — and note that "latest year" is not "latest release" Releases live at `//`, where the year is the **publication** year, not the weather year. Both appear in the path and they are frequently different: `2025/comstock_amy2012_release_2` is a 2025 publication using 2012 weather. Release names follow `__release_`: - `dataset_type` — `resstock` (residential) or `comstock` (commercial) - `weather` — `amy` (actual meteorological year, synchronised across locations) or `tmy3` (typical, **not** synchronised — one location can be in a heat wave while its neighbour is mild) - `release_` — sequence within the publication year. A `release_1.1` form also exists in 2022 and is a revision of `release_1`. `2021/EULP TAG Meeting Presenations/` is a presentation archive, not a release. Filter to `kind == "data_release"` in `catalog.json`. ### The one anomalous release Everything else in this guide applies to the 22 conforming releases. **`2024/resstock_dataset_2024.1/` is the sole exception** and is marked `kind: "anomalous_release"`. **It contains no timeseries data at all** — no `timeseries_aggregates/`, no `timeseries_individual_buildings/`, no `building_energy_models/`. It is annual-results-only: 5.9 TB across 30,055 objects covering 260 upgrade packages, holding `metadata/`, `metadata_and_annual_results/` (by state and national), `weather/`, and `geographic_information/`. Two further traps if you do use it: - **Content is nested one level deeper than every other release**, under `resstock_tmy3/`. The path is `2024/resstock_dataset_2024.1/resstock_tmy3/metadata_and_annual_results/…`. Code assuming `///` finds nothing. - **There is no `upgrades_lookup.json`.** Upgrades are described in `Measure Package Index.csv` using decimal package numbers (`1.01` … `15.28`), while the files are `upgrade1.parquet` … `upgrade260.parquet`. **No crosswalk between the two schemes is published**, so you cannot determine which measure package a given `upgradeN.parquet` represents. Treat the upgrade numbering as unresolved. Its `data_dictionary.tsv` and `enumeration_dictionary.tsv` are present and valid, so column-level interpretation is sound. The key templates in §4 do **not** apply to it. ### Current release per lineage | Lineage | Current | Notes | |---|---|---| | `resstock_amy2018` | `2025/resstock_amy2018_release_1` | Recommended residential AMY | | `resstock_amy2012` | `2025/resstock_amy2012_release_1` | Dictionary is a **provisional copy** from the AMY2018 sibling. Upgrade lookup incomplete | | `resstock_tmy3` | `2024/resstock_tmy3_release_2` | No 2025 TMY3 ResStock release | | `comstock_amy2018` | `2025/comstock_amy2018_release_3` | Recommended commercial AMY | | `comstock_amy2012` | `2025/comstock_amy2012_release_2` | Numbered `release_2` with no `release_1` in this lineage. **Do not use for Hawaii** — see §7 | | `comstock_tmy3` | `2021/comstock_tmy3_release_1` | **2021 only.** Lineage not updated since | > **Trap — silent staleness.** `2025/` contains six releases across four lineages. Choosing "the newest thing under 2025/" gives you a ComStock AMY2012 release when you wanted TMY3, whose only version is from 2021. Resolve by lineage, never by year. **Every current release has at least one results-affecting known issue.** Resolving the lineage is step one; §7 is step two, and it is not optional. --- ## 3. Resolve upgrades before building keys Every release ships `upgrades_lookup.json` at its root: a flat `{"0": "Baseline", "1": "...", ...}`. Upgrade `0` is always the baseline existing stock. ``` /upgrades_lookup.json ``` Upgrade counts range from 1 (2021) to 66 (`2025/comstock_amy2018_release_3`). Upgrade IDs are **not stable across releases** — upgrade 5 in one release is not upgrade 5 in another. Use `measure_name_crosswalk.csv`, where present, to map a measure across releases. > **Trap — the lookup can be wrong.** In `2025/resstock_amy2012_release_1`, `upgrades_lookup.json` defines IDs 0–28, but partitions `upgrade=29` through `upgrade=32` exist on disk. Treat the lookup as a starting point, not as authoritative, and reconcile against the partitions you actually intend to read. > > Some releases also have interior gaps — `2023/comstock_amy2018_release_1` has no upgrade 4; `2024/comstock_amy2018_release_1` has neither 4 nor 30. A gap is not an error to retry. --- ## 4. Key construction Placeholders: `{STATE}` uppercase (`CO`), `{state}` lowercase (`co`), `{u}` the upgrade integer. ### ResStock — `2025/resstock_amy2018_release_1` | Content | Key template | |---|---| | Annual results, per state | `metadata_and_annual_results/by_state/full/csv/state={STATE}/{STATE}_upgrade{u}.csv.gz` | | Annual results, national | `metadata_and_annual_results/national/full/parquet/upgrade{u}.parquet` | | Aggregates | `timeseries_aggregates/by_state/upgrade={u}/state={STATE}/up{u:02d}-{state}-{building_type}.csv` | | Individual timeseries | `timeseries_individual_buildings/by_state/upgrade={u}/state={STATE}/{bldg_id}-{u}.parquet` | | Building model | `building_energy_models/upgrade={u}/bldg{bldg_id:07d}-up{u:02d}.zip` | | Weather | `weather/state={STATE}/{county_gisjoin}_2018.csv` | Upgrades 29–32 do not currently resolve under `timeseries_aggregates/` — see §5.1. Building types: `mobile_home`, `multi-family_with_2_-_4_units`, `multi-family_with_5plus_units`, `single-family_attached`, `single-family_detached`. ### ComStock — `2025/comstock_amy2018_release_3` | Content | Key template | |---|---| | Annual results, per county | `metadata_and_annual_results/by_state_and_county/full/parquet/state={STATE}/county={gisjoin}/{STATE}_{gisjoin}_upgrade{u}.parquet` | | Annual results, per county, basic | `metadata_and_annual_results/by_state_and_county/basic/parquet/state={STATE}/county={gisjoin}/{STATE}_{gisjoin}_upgrade{u}_basic.parquet` | | Annual results, aggregated | `metadata_and_annual_results_aggregates/national/full/parquet/upgrade{u}_agg.parquet` | | Aggregates | `timeseries_aggregates/by_state/upgrade={u}/state={STATE}/up{u}-{state}-{building_type}.csv` | | Individual timeseries | `timeseries_individual_buildings/by_state/upgrade={u}/state={STATE}/{bldg_id}-{u}.parquet` | | Building model | `building_energy_models/upgrade={u:02d}/bldg{bldg_id:07d}-up{u:02d}.osm.gz` | | Weather | `weather/amy2018/{county_gisjoin}_2018.csv` | ComStock offers a `basic/` alongside `full/` under `metadata_and_annual_results*`; ResStock offers only `full/`. `basic` is a reduced column set — check `in_basic_metadata_file` in the data dictionary — and its filenames carry a `_basic` suffix that `full` does not. Note the ComStock annual-results path carries **two extra partition levels** (`state=`/`county=`) that the ResStock equivalent does not, and repeats both values in the filename. Verified examples: ``` …/2025/resstock_amy2018_release_1/timeseries_aggregates/by_state/upgrade=0/state=CO/up00-co-single-family_detached.csv …/2025/comstock_amy2018_release_3/timeseries_aggregates/by_state/upgrade=0/state=CO/up0-co-fullservicerestaurant.csv …/2025/resstock_amy2018_release_1/timeseries_individual_buildings/by_state/upgrade=0/state=AK/100102-0.parquet …/2025/comstock_amy2018_release_3/metadata_and_annual_results/by_state_and_county/full/parquet/AK_G0200130_upgrade0.parquet ``` --- ## 5. The two silent-failure traps ### 5.1 ResStock aggregates for upgrades 29–32 are temporarily in the wrong place **This is a known publication defect with a fix in progress — check `manifest.json` before coding around it.** In `2025/resstock_amy2018_release_1` and `2025/resstock_amy2012_release_1`, `upgrades_lookup.json` defines upgrades 0–32, but `timeseries_aggregates/` contains **only 0–28** across all four geographies. Aggregates for upgrades 29–32 were written to a temporary location, `timeseries_aggregates_new_env/`. Requesting upgrade 30 from `timeseries_aggregates/` produces a key that does not exist — **no data, no exception, no warning**. These aggregates are being **regenerated under `timeseries_aggregates/upgrade={id}/`** in the standard convention, after which the temporary tree will be deleted. Once that lands, the normal §4 key template covers the full upgrade range and no special handling is needed. **Until then**, upgrades 29–32 are reachable only at: ``` timeseries_aggregates_new_env/by_state/upgrade={u}/state={STATE}/up{u:02d}-{state}-{building_type}.csv ``` Do **not** hard-code this path — it is temporary and will be removed. Resolve it from the `MISPLACED_AGGREGATES` warning in the release manifest, and prefer to retry the standard path first: ```python def aggregate_prefix(upgrade_id: int) -> str: # Standard location. Falls back to the interim tree only while the # MISPLACED_AGGREGATES warning is present in the release manifest. return "timeseries_aggregates" ``` The interim tree also **mixes unchunked CSV with chunked Parquet**: ``` up30-ak-single-family_detached.csv up30-ak-single-family_detached_chunk_1_of_4.parquet ``` Reading one chunk gives you a quarter of the data with no indication anything is missing. Match `_chunk_(\d+)_of_(\d+)` and concatenate all `n`, or use the CSV. The regenerated data will be unchunked CSV only. ### 5.2 Upgrade padding is inverted between ResStock and ComStock — but only below upgrade 10 | Upgrade | ResStock aggregate filename | ComStock aggregate filename | |---|---|---| | 0 | `up00-co-…` | `up0-co-…` | | 9 | `up09-co-…` | `up9-co-…` | | 10 | `up10-co-…` | `up10-co-…` | The conventions **agree at and above upgrade 10 and disagree below it**. Code tested on upgrade 10 passes, then silently returns nothing for the baseline. Because the baseline is the most commonly requested scenario, this trap tends to surface in production rather than in testing. ```python def upgrade_prefix(dataset_type: str, upgrade_id: int) -> str: return f"up{upgrade_id:02d}" if dataset_type == "resstock" else f"up{upgrade_id}" ``` --- ## 6. Upgrade ID rendering, all five forms The same upgrade is written five different ways within one release. Never reuse a rendering across directories. | Location | ResStock | ComStock | |---|---|---| | Partition directory, timeseries | `upgrade=0` | `upgrade=0` | | Partition directory, models | `upgrade=0` | `upgrade=00` | | Aggregate filename | `up00-` | `up0-` | | Model filename | `-up00.zip` | `-up00.osm.gz` | | Individual timeseries filename | `-0.parquet` | `-0.parquet` | | Annual results filename | `_upgrade0.csv.gz` | `_upgrade0.parquet` | Note the two that differ *within* ComStock: the partition directory is padded for models and unpadded for timeseries. `bldg_id` is likewise rendered two ways: **zero-padded to 7** in model filenames (`bldg0000001-…`) and **unpadded** everywhere else (`100102-0.parquet`). --- ## 7. Known issues **Do not read this section as the list.** It tells you how to obtain the list, because the list changes and a copy in a guide goes stale. There are 32 known issues across the repository; 13 affect published results. ### Get them programmatically Every release manifest carries the issues affecting that release, so one fetch answers the question for the release you are using: ```python man = json.loads(s3.get_object(Bucket=B, Key=PREFIX + "manifest.json")["Body"].read()) blocking = [i for i in man["known_issues"] if i["affects_results"]] if blocking: for i in blocking: print(i["id"], i["severity"], i["summary"], i["detail_url"]) ``` The same information is in `catalog.json` as `known_issues[]`, keyed to releases by `releases[].known_issue_ids` — use that when comparing releases before choosing one. **Check `warnings` too.** A release whose results are affected carries a `KNOWN_ISSUES_AFFECT_RESULTS` warning as its **first** warning. If you check only one field, check that one: ```python if any(w["code"] == "KNOWN_ISSUES_AFFECT_RESULTS" for w in man["warnings"]): ... # stop and read known_issues before using the numbers ``` ### Fields you will need | Field | Meaning | |---|---| | `affects_results` | `true` means the **numbers** are affected, not merely their documentation. Treat as blocking until you have read the detail | | `severity` | `critical`, `significant`, `caution`, `informational` | | `status` | `open`, `resolved`, `partially_resolved`, `wont_fix`, `fix_in_progress`, `inherent`. `wont_fix` means the workaround **is** the answer — no corrected release is coming | | `source` | `upstream` means the entry is a summary of an authoritative ResStock or ComStock page; `detail_url` has the full text, magnitudes and workarounds | | `detail_url` | Follow it before acting. The summaries here are deliberately short | `KNOWN_ISSUES.md` at the repository root carries the prose version, with worked examples for the issues that need one. ### What affects results in the releases you are most likely to use Generated from `catalog.json`; current as of the `status_date` in that file. | Release | Issue | What it does | |---|---|---| | `2024/resstock_tmy3_release_2` | `resstock-2024r2-electricity-energy-bills` | Electricity and energy bills were computed from three different rate sets, so bills are internally inconsistent and bill SAVINGS can be nonsensical -- a bill can rise while consumption falls. | | ″ | `resstock-2024r2-measure-applicability` | Dual-fuel heat pump measures (upgrades 4, 9 and 14) are marked not-applicable for a subset of models that should have received them, so their upgrade results equal the baseline. | | ″ | `resstock-wall-type-maine` | Maine wall types are mis-characterised in the source assessor data, showing far more masonry and far less wood framing than local experts expect. | | `2025/comstock_amy2012_release_2` | `comstock-2025r2-package-naming` | Results for package 6 (Demand Flexibility) and package 9 (Geothermal Heat Pump) were swapped. | | ″ | `comstock-2025r2-hawaii-amy2012-weather` | The AMY2012 weather files for Hawaii contain temperatures below freezing, which does not occur there. | | `2025/comstock_amy2018_release_3` | `comstock-2025r3-california-swh` | Service water heating was not modeled in any California buildings due to a workflow error. | | `2025/resstock_amy2012_release_1` | `resstock-2025r1-electric-panel-capacity` | Electric panel capacity outputs are WRONG where an HVAC measure is a like-for-like replacement: the NEC load calculation counts the replacement as a new load, so panel constraints are overstated. | | ″ | `resstock-wall-type-maine` | Maine wall types are mis-characterised in the source assessor data, showing far more masonry and far less wood framing than local experts expect. | | `2025/resstock_amy2018_release_1` | `resstock-2025r1-electric-panel-capacity` | Electric panel capacity outputs are WRONG where an HVAC measure is a like-for-like replacement: the NEC load calculation counts the replacement as a new load, so panel constraints are overstated. | | ″ | `resstock-wall-type-maine` | Maine wall types are mis-characterised in the source assessor data, showing far more masonry and far less wood framing than local experts expect. | 10 results-affecting issues across the 5 current releases. Every release also carries issues that do not affect results — read the full `known_issues` array in its manifest rather than only this table. **Two patterns worth internalising.** A results-affecting issue is not always a wrong number in the column you are reading — `comstock-2025r3-package-naming` and `comstock-2025r2-package-naming` are *label* swaps, so the results are right and the name attached to them is wrong. Resolve upgrade identity from `upgrades_lookup.json`, never from a name embedded in a results file. And several issues are scoped by geography rather than by column: Hawaii in ComStock AMY2012, California in several ComStock releases, Maine in every ResStock release. A national aggregate can be sound while the state you actually care about is not. --- ## 8. Reading the data **Column prefixes.** `in.` baseline characteristic, `out.` simulation output, `upgrade.` post-measure characteristic where it differs from baseline. **Dictionaries.** Every release carries a dictionary on one standardized schema: `field_name, field_location, data_type, units, field_description, allowable_enumerations`, plus `in_full_metadata_file` and `in_basic_metadata_file` for ComStock. The 2025 releases carry it as `data_dictionary.tsv`; earlier releases keep their original file unchanged and add `data_dictionary_standardized.tsv` beside it — prefer the standardized one. Both open with a provenance comment, so read with `comment='#'`: ```python dd = pd.read_csv(key, sep=" ", comment="#") ``` `field_name` matches the Parquet column exactly, including any trailing `..unit`. Two caveats: parse by header name rather than position, because the untouched pre-2025 originals still carry seven different layouts; and a dictionary whose provenance line says `status=provisional` was copied from a sibling release and is indicative, not authoritative (`standardized_schema: false` in `catalog.json` marks the same thing). **Units.** Energy is kWh. Verify against the dictionary for the specific release — units were corrected in the 2021 datasets in March 2022, so older cached copies may disagree with the current files. **Timestamps.** Aggregate and timeseries timestamps are **15-minute period-ending in EST**: `2018-01-01 00:30:00` covers 00:15:00–00:30:00 EST. Weather file timestamps are **local standard time** — a different basis. Do not join the two without converting. **Aggregate columns.** `models_used` is the number of simulated models in the aggregate. `units_represented` (residential) or `floor_area_represented` in ft² (commercial) is what the aggregate scales to. To reproduce an aggregate's membership, filter the annual-results file on the same geography and building type. **Geographic identifiers.** Counties and PUMAs use GISJOIN identifiers (`G0200130`); states use two-letter postal abbreviations. The census-tract lookup is in `geographic_information/`, but **the filename differs by dataset**: ResStock 2025 ships `spatial_tract_lookup_table.csv`, ComStock 2025 ships `spatial_tract_lookup_table_publish_v10.csv`. ComStock additionally ships `map_of_pumas_whole_us.geojson`, which ResStock does not. Resolve the filename by listing the directory — it holds fewer than ten objects, so this is the one place listing is cheap. --- ## 9. Minimal working example ```python import boto3, pandas as pd, io s3 = boto3.client("s3", region_name="us-west-2") BUCKET = "oedi-data-lake" ROOT = "nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/" REL = "2025/resstock_amy2018_release_1/" # resolve by lineage, not by year def agg_key(upgrade, state, building_type, dataset_type="resstock", tree="timeseries_aggregates"): # tree defaults to the standard location. Override only for upgrades 29-32 # in the 2025 ResStock releases, until the regeneration in KI-001 lands. up = f"up{upgrade:02d}" if dataset_type == "resstock" else f"up{upgrade}" return (f"{ROOT}{REL}{tree}/by_state/upgrade={upgrade}/state={state.upper()}/" f"{up}-{state.lower()}-{building_type}.csv") key = agg_key(0, "CO", "single-family_detached") body = s3.get_object(Bucket=BUCKET, Key=key)["Body"].read() df = pd.read_csv(io.BytesIO(body)) print(df["models_used"].iloc[0], "models in this aggregate") if df["models_used"].iloc[0] < 1000: print("WARNING: below the 1,000-model guidance; treat conclusions as indicative") ``` --- ## 10. Checklist before trusting a result 1. Release resolved **by lineage**, not by the newest year prefix. 2. Release manifest checked for a `KNOWN_ISSUES_AFFECT_RESULTS` warning, and every `known_issues` entry with `affects_results: true` read and its `detail_url` followed. 3. Upgrade ID confirmed to exist **in the specific directory** you are reading, not just in `upgrades_lookup.json`. 4. For ResStock upgrades 29–32, the release manifest checked for a `MISPLACED_AGGREGATES` warning, and any `_chunk_i_of_n` files concatenated if the interim path is still in use. 5. Upgrade prefix padded correctly for the dataset type (`up00` ResStock, `up0` ComStock). 6. Dictionary read with `comment='#'`, parsed **by header name** never by position, and checked for `status=provisional` in its provenance line. 7. `models_used` inspected against the 1,000-model guidance. 8. Timestamp basis reconciled — EST period-ending for results, local standard time for weather. 9. Geographic scope checked against §7: Hawaii, California and Maine each have issues confined to one state, which a national total will hide.