14 Add Nudging decay (including reading LastObs files) #310

jhrehanoaa · 2021-05-06T21:08:30Z

creates the last obs dataframe.
sets a true/false condition in the yaml to use either fresh start last obs file to create da decay predictions or run from our models final discharge values.
if no discharge values from routelink are matching it stop the run until this condition is fixed.
still need to generalize some of the code further.

… various input sizes

src/python_framework_v02/troute/nhd_io.py

groutr · 2021-05-10T18:42:11Z

src/python_framework_v02/troute/nhd_io.py

+        prediction_df = pd.DataFrame(index=model_discharge_last_ts.index)
+
+        for time in range(0, 720, 5):
+            weight = np.exp(time / -120)


Probably better to use math.exp here. The numpy operations are optimized to work over arrays and have a high overhead when fed scalars.

the weight is eventually applied to an array and it might be efficient to stack the computation into the array operation and perhaps take advantage of numpy optimizations. Just thinking...

src/python_framework_v02/troute/nhd_io.py

…side equation

… to print values to identify if da is properly working

…efore

This reverts commit 0455466.

jameshalgren · 2021-05-25T23:23:45Z

Fix Diffusive function signature to include last_obs
(test with python compute_nhd_routing_SingleSeg_v02.py -f ../../test/input/yaml/Florence_Benchmark_diff.yaml)

jameshalgren · 2021-05-26T14:58:45Z

src/python_framework_v02/troute/nhd_io.py

+        #                 delta["last_nudge"] + model_discharge_last_ts["model_discharge"]
+        #             )
+        # prediction_df["0"] = model_discharge_last_ts["model_discharge"]
+        return final_df


@jhrehanoaa

Need to return last_obs_df and last_obs_time_df.

…y and changing parity check even when all uses are turned off

…-route into lastobs_using_model_or_file

To facilitate this, added a constant qvd_ts_w (flowveldepth timestep width) to define the standard column width.

* added last obs df including discharge and ids * lastobs nc file folder * formatting and cleanup * added both discharge and model discharge * added prediction delta timesteps * da decay with exp decay incorporated, unblackened for readability * added check between obs file results and our fvd output * removed pdbs and set to run on lastobs * added verbose statements * restructuring of lastobs df to simplify process * generalized last timestep index call to automatically determine from various input sizes * github requested fixes to if statements and cleanup * updated variable * working ncar da decay prototype need to bring in real lastobs data inside equation * getting da values in order through classic da assim technique * pushing ids and values to mc reach * fixed gage id matching, cython is broken cant compile correctly, need to print values to identify if da is properly working * saving changes, trying to fix old DA function * restructing da timeslice file read to use datetime, not generalized before * added last obs df including discharge and ids * lastobs nc file folder * formatting and cleanup * added both discharge and model discharge * added prediction delta timesteps * da decay with exp decay incorporated, unblackened for readability * added check between obs file results and our fvd output * removed pdbs and set to run on lastobs * added verbose statements * restructuring of lastobs df to simplify process * generalized last timestep index call to automatically determine from various input sizes * github requested fixes to if statements and cleanup * updated variable * generalized da naming conventional and date timeframe with improved interpolation * removed extra comments * remove dependence on not-yet-created flowveldepth * name "last_obs_file" * include data_assimilation_parameters to yaml * include empty dict for data_assimilation_parameters in yaml * black * added paths to shared drive locations and blackened files * quick merge changes * files working * sync for merge * sync for merge * sync merge to upstream * move last obs function next to usgs_da function * add TODOs * add lastobs to other parallel modes * move last_obs above usgs_df * cimports and cdefs for last_obs * fixed broken usgs_df names were changed to folder in some areas and filter in others * moved da functions into single wrapper in nhd_network_utilities also harmonized inputs a bit. * add da yaml file (DATA NOT YET ADDED) * drop non-5-minute entries from time_slices Also harmonizing inputs for merge. * add function for finding the tailwater for a given segment * add masks * use pandas date_range * cleanup * add comment showing possible handling of extended DA * Revert "add comment showing possible handling of extended DA" This reverts commit 0455466. * temporarily disable last_obs * Update example yaml with inputs that work * temporarily disable last_obs * update comment * adjust DA for perfect match * removed filter list * use efficient shape call for usgs_positions_list length * add gage_maxtime and pseudocode for lastobs * Identified major hard-code issue in structured- and structured-obj * update lastobs comments and pseudocode * update da test yaml file with additional gage options * use "reindex" to fill/eliminate columns for usgs_df * functions in place for decay but last obs file is behaving incorrectly and changing parity check even when all uses are turned off * added decay timestep count * Use new fields in flowveldepth to simplify initial condition handling To facilitate this, added a constant qvd_ts_w (flowveldepth timestep width) to define the standard column width. * add two additional segments for parity checking * reconfigure reach splitting to consider gages * update diffusive call signature * yaml updates for test * black Co-authored-by: James Halgren <[email protected]>

jhrehanoaa added 11 commits April 30, 2021 21:38

added last obs df including discharge and ids

4c414cb

lastobs nc file folder

3b9520d

formatting and cleanup

4b71387

added both discharge and model discharge

53de784

added prediction delta timesteps

4355e98

da decay with exp decay incorporated, unblackened for readability

90ed41a

added check between obs file results and our fvd output

1e724c8

removed pdbs and set to run on lastobs

536754d

added verbose statements

f07a3bf

restructuring of lastobs df to simplify process

148ce6c

generalized last timestep index call to automatically determine from…

578f555

… various input sizes

groutr reviewed May 10, 2021

View reviewed changes