GPS blending does not work in EKF2 replay

[CarlOlsson]

Describe the bug
Based on this comment it looks like EKF replay has been used in the past to develop the GPS blending algorithm. However, now it seems like it is not working as it should.
Here are the blend weights (added to estimator_status for logging) on a vehicle with two GPS modules connected in a hand held test. Top plot is original logfile, bottom is replayed

[CarlOlsson]

Replayed logfile at the top, original logfile at the bottom

The issue is that even though all GPS data is logged without dropouts the ekf2_timestamp topic can have some dropouts. And if this happens when a GPS measurement is being fused a timeout is triggered in ekf2_main

https://github.com/PX4/Firmware/blob/e0f016c2b3db160284bbaaadaf70bef84592d81e/src/modules/ekf2/ekf2_main.cpp#L1844-L1847

[CarlOlsson]

@priseborough As mentioned above, when replaying data I found the following issue with the GPS blending logic:

If one GPS module timeouts ekf2 instantly switches out of blending and pushes the other GPS data directly into the EKF, resulting in high innovations

I remember you mentioning that you were concerned about GPS glitches when discussing low pass filtering the GPS blending weights. So my questions is, is this by design that you want to instantly switch out of blending?

[CarlOlsson]

@bresch The original issue reported here (regarding replay) should be solved by #14114 and making sure that there are very few logging dropouts in the logfile (<0.15%). I have noticed that in order to achieve this CPU load needs to be below 70% on fmu-v4.

Here is diff(number_of_times_published) for CPU load ~65% top and ~72% bottom

However, the fact that the output of the blending algorithm step changes if one module times out can still be regarded as an issue but lets discuss that on Tuesday