EIA profile-based powerplant age imputation#31
Conversation
…ine based on dated capacities.
for more information, see https://pre-commit.ci
|
Adding comment. The new method is only applied to assets with the 'operating' status. It is not yet applied to historical (retired) nor future statuses (pre-construction, construction, announcements). Once you are happy with the methodology as a concept, I can extend it as these other categories will require slightly different target profiles e.g. a linear forecast for the future ones. |
| # Harmonise powerplant categories with the category names used by the | ||
| # annual reference-capacity dataset. | ||
| REFERENCE_CATEGORY_MAP = {"fossil": "fossil fuels", "bioenergy": "biomass and waste"} | ||
|
|
There was a problem hiding this comment.
This is a duplicate of EIA_CAT_MAPPING in scripts/_utils.py.
Please use that mapping (should work just fine in all cases).
| HISTORICAL = {"operating", "retired"} | ||
| CURRENT = {"operating"} | ||
| SCENARIO_MAP = { | ||
| "historical": HISTORICAL, | ||
| "construction": HISTORICAL | {"construction"}, | ||
| "pre_construction": HISTORICAL | {"construction", "pre-construction"}, | ||
| "announced": HISTORICAL | {"construction", "pre-construction", "announced"}, | ||
| } |
There was a problem hiding this comment.
We should only have one source of truth in the code.
CURRENT has some overlap with HISTORICAL.
We likely need a better data structure to handle this.
| def _save_age_imputation_diagnostics( | ||
| diagnostics: pd.DataFrame, output_path_str: str | ||
| ) -> None: | ||
| """Save row-level age-imputation diagnostics if requested.""" | ||
| output_path = Path(output_path_str) | ||
| output_path.parent.mkdir(parents=True, exist_ok=True) | ||
| diagnostics.to_parquet(output_path, index=False) | ||
|
|
||
|
|
||
| def _save_age_profile_diagnostics( | ||
| diagnostics: pd.DataFrame, output_path_str: str | None | ||
| ) -> None: | ||
| """Save annual age-imputation profile diagnostics.""" | ||
| if output_path_str is None: | ||
| return | ||
|
|
||
| output_path = Path(output_path_str) | ||
| output_path.parent.mkdir(parents=True, exist_ok=True) | ||
| diagnostics.to_parquet(output_path, index=False) |
There was a problem hiding this comment.
Both of these are unnecessary.
- Snakemake is in charge of doing
mkdir. main()should be in charge of all file creation decisions.
| return result | ||
|
|
||
|
|
||
| def _impute_remaining_start_years( |
There was a problem hiding this comment.
We should consider whether the "group_average" method is even justified any more. Looking at the results, there should be no situation in which this would be preferable to the capacity profile. If so, we should just replace with method == "drop" for over-conservative cases.
Can be a separate issue / PR.
| if method == "group_average": | ||
| return (_impute_start_years_by_group_average(prepared_df), pd.DataFrame()) | ||
|
|
||
| if method == "capacity_profile": | ||
| return _impute_start_years_by_capacity_profile( | ||
| prepared_df, | ||
| reference_capacity_df=reference_capacity_df, | ||
| lifetimes=lifetimes, | ||
| ) | ||
|
|
There was a problem hiding this comment.
Are the returned diagnostics really necessary?
A lot of code is dedicated / branching is dedicated to them.
group_average is even producing a dummy to comply with them, suggesting an anti-pattern.
| if relocated_gdf.empty: | ||
| imputed = relocated_gdf | ||
| _save_age_imputation_diagnostics(pd.DataFrame(), age_imputation_output_path) | ||
| _save_age_profile_diagnostics(pd.DataFrame(), age_profile_output_path) |
There was a problem hiding this comment.
Move this check to main and remove the functions.
That function "owns" file creation. If we must keep diagnostics, just make impute return a tuple.
There was a problem hiding this comment.
Seems like the plotting functions do not respect this either, but we should strive to do this to keep the code tidier.
| diagnostics = _build_age_imputation_diagnostics( | ||
| original_df=original, | ||
| aged_df=imputed, | ||
| status_final=status_final, | ||
| start_year_source_type=start_year_source_type, | ||
| end_year_source_type=end_year_source_type, | ||
| lifetimes=lifetimes, | ||
| start_year_imputation_method=start_year_imputation_method, | ||
| ) |
There was a problem hiding this comment.
These diagnostics might be more justified than the profile ones, but we should still consider if they are really needed. They are resulting in a lot of duplicated data and extended lines of code. It's OK to save some diagnostics, but only the minimum necessary.
| countries = sorted(profile_df["country_id"].dropna().unique()) | ||
| n_countries = len(countries) |
There was a problem hiding this comment.
dropna is not necessary (and may silently let issues pass)
| countries = sorted(profile_df["country_id"].dropna().unique()) | |
| n_countries = len(countries) | |
| countries = sorted(profile_df["country_id"].unique()) | |
| n_countries = len(countries) |
| observed_color = cmap(0) | ||
| imputed_color = cmap(1) |
There was a problem hiding this comment.
You can just remove this and instead use color=tab20:blue or color=tab20:cyan below.
2 participants
Fixes #
Summary of changes in this pull request
Adds a deterministic alternative to group-average powerplant age imputation.
The new method:
Validation
Tested on:
Notes
Powerplant categories are harmonised with the reference statistics categories for:
bioenergy→biomass and wastefossil→fossil fuelsReviewer checklist
pipdependencies in the module's environment files (workflow/envs/).pathvars(e.g.,<results>) in their inputs and outputs.pre-commit.citests pass.INTERFACE.yamlmentions all relevantpathvarsandwildcards.README.mddescribes how to use the module and has the necessary citations.