[1pt] PR: Rating curves adjustments with ml-bathymetry data and optimized mannN

[hhs732]

This PR focuses on adjusting rating curves by using bathymetric data and optimized channel roughness values. The bathymetry data includes eHydro surveys and AI-based datasets created for all NWM streams. New manning roughness values were developed for each feature-id using a differential evolution objective function (OF). The OF minimizes the number of the false_positives and false_negatives cells in our flood inundation maps where we have test cases across the CONUS.

Even though the Python scripts of roughness manning number optimization were not included in this branch, optimized roughness values can be found here: /fim-data/inputs/rating_curve/variable_roughness/mannings_optz_fe_clusters_so3.csv. Detailed python scripts also can be found here: /fim-data/outputs/heidi-mannN-optimization/projects/bathy_mannN_projects/dev-bathymetric-adjustment-mannN-optz/.

Changes

• src/bathymetric-adjustment.py: correct_rating_for_ai_based_bathymetry function was added to the script. This function processes AI-based bathymetry data and adjusts rating curves using this data. Also apply_src_adjustment_for_bathymetry function was added to prioritize USACE eHydro over AI-based bathymetry dataset. The multi-processing function multi_process_hucs was updated based on the latest code. Also, an ai_toggle parameter was added to apply_src_adjustment_for_bathymetry and process_bathy_adjustment functions. When ai_toggle = 1, The SRCs will be adjusted with the ai_based bathymetry data. the default value for ai_toggle = 0, means no ai_based bathy data is included.

• src/bash_variables.env: New variables and their paths were added. Also, a new input file with the nwm feature_ids and optimized channel roughness and overbank roughness attributes was created and stored here:
  /fim-data/inputs/rating_curve/variable_roughness/mannings_optz_fe_clusters_so3.csv
  The locations of these files were also added to the bash_variables.env.
  Please note that when ai_toggle = 1, the manning roughness values should be switched to vmann_input_file=${inputsDir}/rating_curve/variable_roughness/mannings_optz_fe_clusters_so3.csv in the current version.

Here is a list of new/updated input files:

1. /fim-data/inputs/rating_curve/variable_roughness/mannings_optz_fe_clusters_so3.csv
   This CSV file contains the new optimized roughness values. It will replace this file:
   vmann_input_file=${inputsDir}/rating_curve/variable_roughness/mannings_global_nwm3.csv

2. bathy_file_aibased=${inputsDir}/bathymetry/ml_outputs_v1.01.parquet
   This file contains the ml-bathymetry and manning roughness values data.

3. bathy_file_ehydro=${inputsDir}/bathymetry/final_bathymetry_ehydro.gpkg
   We already had this file, the name of the variable has changed from bathymetry_file to bathy_file_ehydro, and it was updated.

• fim_post_processing.sh: New arguments were added. Please note that the default value for ai_toggle = 0 is included here.

Testing

This PR has been tested on 11 HUC8s around the Illinois River, Ohio River, Meramec River, etc.

UPDATE: Jan 8, 2025: Please run a test for the multiple configurations, ie) with AI turned on/off, bathy on/off, etc.

Deployment Plan (For developer use)

How do the changes affect the product?

• Code only?
• If applicable, has a deployment plan be created with the deployment person/team?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set?
• Has new or updated python packages?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

[RobHanna-NOAA]

Ah. Looks like we have new inputs (and/or) some updated ones, correct?

[hhs732]

Hi Rob, yes that is correct.

…

On Fri, Nov 8, 2024 at 1:56 PM Rob Hanna - NOAA ***@***.***> wrote: Ah. Looks like we have new inputs (and/or) some updated ones, correct? — Reply to this email directly, view it on GitHub <#1340 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFRLX2UVV2DFIDLD565RASDZ7UXP5AVCNFSM6AAAAABROJK2XCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRVHAYTAOJSGM> . You are receiving this because you were assigned.Message ID: ***@***.***>

[RobHanna-NOAA]

ok. awesome. Can you make a list of which files/folders so I know where to get them and get them moved around. Is it ok for me to start copying them around now? or not quite yet?

[hhs732]

Hi Rob,
There are three input files that I added/changed:

1. /fim-data/inputs/rating_curve/variable_roughness/mannings_optz_fe_clusters_so3.csv
   This CSV file contains the new optimized roughness values. It will replace this file:
   vmann_input_file=${inputsDir}/rating_curve/variable_roughness/mannings_global_nwm3.csv

2. bathy_file_aibased=${inputsDir}/bathymetry/ml_outputs_v1.01.parquet
   This file contains the ml-bathymetry and manning roughness values data.

3. bathy_file_ehydro=${inputsDir}/bathymetry/final_bathymetry_ehydro.gpkg
   We already had this file, the name of the variable has changed from bathymetry_file to bathy_file_ehydro, and it was updated.

[mluck]

The new files have been uploaded to all of the environments.

[ZahraGhahremani]

Please merge in the current dev branch.

[ZahraGhahremani]

Slide deck, comparing this FB with dev for May 10th flood event.

https://docs.google.com/presentation/d/1QogJN1w8jG-LqOyQ_yM-lCIqO3iuNiQ2/edit?usp=sharing&ouid=104281778160768065124&rtpof=true&sd=true

[RobHanna-NOAA]

Based on a chat with Carson today, a slight change of plans and code change is require:

• Have a new switch to turn on the new aibased bathy, but still continue use ehydro which was there previously. We already have a params_template.env for bathymetry_adj, and will want an additional one to turn off/on for aibased bathy. The default for this new switch is to have it turned off.

• When the aibased is turned off, we want to use the original variable_roughness/mannings_global_nwm3.csv. If it is turned on, then we want to use the new variable_roughness/mannings_optz_fe_clusters_so3.csv. That might require having two bash_variable depending on which is being used.

• At this point, the plan is to leave the new bathy functionality aibased feature turned off for next springs 2.2.0 production version. However, Carson will talk to Fernando and Derek to confirm if it will be on/off at that time. For now.. we will leave that part turned off and can turn it on at last minute if desired.

• Once we add this new "on/off" switch, this code can be merged into our code base. No need to hold off merging it indefinitely.

[hhs732]

understood. I am starting to implement those now.

[RobHanna-NOAA]

For the next approver (retest)... Please run a test for the multiple configurations, ie) with AI turned on/off, bathy on/off, etc. :)

[hhs732]

ai_toggle was added. @ZahraGhahremani would you please test it again with ai_toggle = 0 and ai_toggle = 1? Thank you.

[ZahraGhahremani]

I tested for minor changes in toggle and it worked as expected.