Parameter system enhancement #14363

RomanBapst · 2020-03-11T09:13:15Z

This pull request brings the following enhancements to the PX4 parameter system:

Short summary

ability to store and fetch the true default parameter value which may stem from the generic defaults, board config, vehicle_type config (mc, fw, vtol) or the airframe config.
extracted parameters from autostart scripts and put them into dedicated parameter files, which are converted to bson format at build time
made the SYS_AUTOCOFNIG mechanism obsolete

Detailed Summary

Introduction of param load-as-default param_file. This new command reads parameter values from a file and stores them internally as the default value for that parameter. This default value is not the same as the actual parameter value, the two things are stored separately. If that parameter has not been modified before (e.g. it does not exist in the list of params stored in FRAM) it will also set the value of the parameter accordingly.
This command is used at each place where there used to be the AUTOCNF section.
In order to make use of the above command all the parameters being set in the autoconfig sections had to be moved to appropriate files which are now used to generate bson format parameter files.
This brings the advantage that the default board-, vehicle_type- and airframe config parameters are now accessible as files.
order of loading default parameters: The current order of loading default params is:

board defaults
vehicle type defaults
airframe config default

I created a spreadsheets which shows how the actual- and default value of a specific parameter changes for different scenarios.
https://docs.google.com/spreadsheets/d/1RVHwuwpJRBwl1YKYZ88FEJS0VzTK17DghgDgVqUuJwc/edit?usp=sharing

Open points for discussion

there are parameters which are set in various autostart scripts but outside of any AUTOCNF section. Those parameter will appear as modified parameters.
calibration parameters will also show up as modified parameters

dagar · 2020-03-12T16:03:28Z

extracted parameters from autostart scripts and put them into dedicated parameter files, which are converted to bson format at build time

Is there any actual benefit to keeping this as bson other than import simplicity? Wondering if we should do it directly in a human readable format and leave the possibility of loading from the sd card open.

dagar · 2020-03-12T16:10:16Z

I'm wondering if the safest thing to do would be to load the defaults as soon as possible right next to the param load.

https://github.com/PX4/Firmware/blob/9162c8b196785e7452c928e199118b441e68f320/ROMFS/px4fmu_common/init.d/rcS#L139-L146

What do you think?

dagar · 2020-03-12T16:11:15Z

src/lib/parameters/parameters.cpp

 		param_wbuf_s *s = param_find_changed(param);

-		if (s == nullptr) {
+		// // don't store new value if default
+		// if (val_is_default) {


Why is this commented?

@dagar I was thinking about one case where this might lead to undesired behavior but thinking about it again I think it's ok. I will add this again.

BazookaJoe1900 · 2020-03-12T21:35:05Z

I liked the this approach! and have some small notes/questions:

what about renaming the airframe defaults files from 10015.params to 10015_default.params? so it can be clearer its defaults?
there are many return 0/-1. that can be changed to return PX4_OK/PX4_ERROR
is there a way to avoid the 'static bool set_as_default' and in some way to move it as argument to 'param_import_callback'? its just confuses...

hamishwillee · 2020-03-12T21:39:45Z

@RomanBapst @dagar Does this change put us in a position where parameters can be translated?
Specifically can we round-trip the generated files through crowdin and use them to export parameter docs and whatever is used by QGC?

RomanBapst · 2020-03-16T07:23:15Z

I'm wondering if the safest thing to do would be to load the defaults as soon as possible right next to the param load.

https://github.com/PX4/Firmware/blob/9162c8b196785e7452c928e199118b441e68f320/ROMFS/px4fmu_common/init.d/rcS#L139-L146

What do you think?

Yes, I think we can do that. Let me change and do some tests.

RomanBapst · 2020-03-16T07:26:05Z

@BazookaJoe1900

what about renaming the airframe defaults files from 10015.params to 10015_default.params? so it can be clearer its defaults?

Yes, I think we can do that.

there are many return 0/-1. that can be changed to return PX4_OK/PX4_ERROR

Ok, will check.

is there a way to avoid the 'static bool set_as_default' and in some way to move it as argument to 'param_import_callback'? its just confuses...

Possibly, I agree that the current solution is not the best.

RomanBapst · 2020-03-16T07:31:08Z

@hamishwillee

Does this change put us in a position where parameters can be translated?
Specifically can we round-trip the generated files through crowdin and use them to export parameter docs and whatever is used by QGC?

I can explain to you what we have right now. The generic PX4 defaults are still and only accessible via the Parameters.xml file.
For each board (fmuv2, fmuv3, ...) there is a parameter file holding the board defaults.
There are 3 parameters files holding default values for each multicopter, fixed wing and vtol.
For each airframe there is a paramter file.

The question is in which format do you want to have them. Before build time they are available in a human readable format and after build time they are available in bson format (the format which the PX4 paramter system uses to load and save params.)

- added param import-as-default command - support storing a default parameter value set at run-time Signed-off-by: RomanBapst <[email protected]>

Signed-off-by: RomanBapst <[email protected]>

human readable parameter files Signed-off-by: RomanBapst <[email protected]>

Signed-off-by: RomanBapst <[email protected]>

…efault Signed-off-by: RomanBapst <[email protected]>

Signed-off-by: RomanBapst <[email protected]>

bys1123 · 2020-04-18T21:33:05Z

Is this PR can also save flash memory?

hamishwillee · 2020-07-28T00:59:40Z

FYI We are in the process (see #15389) of adding support for GCS to query a vehicle for metadata and other information. The GCS requests metadata of a particular type and COMPONENT_INFORMATION message is returned with URI (HTTP or mavftp) of file containing that data in .json.gz format.

There has been some discussion that it would be useful to be able to provision parameters as a file via MAVFTP and for GCS request them - a compressed .json.bz file likely to be more far efficient to download via MAVFTP than via the current parameter format.

Don't know if this would impact how you implement this.

FWIW Very cool that the defaults can be supplied as a file. Allowing these to be on SD card or in firmware and have the system choose the SD card preferentially is a good model. Having the default be factory updatable via mavlink even cooler.
Any reason you can't have the file format as either json or json.gz and the system accepts either (or json and bson)?

dagar · 2020-08-25T14:05:32Z

I'd like to get this moving again. Can we establish a plain text parameter file format (or use QGCs) and skip the build time bson generation?

My goal is that ultimately an airframe is nothing more than a set of parameters. Some of those options can live directly in the code base for convenience, but for the majority of users it should be sufficient to have a simple parameter file they can save/load from QGC.

dagar · 2020-08-25T14:09:12Z

Why do we need both import-as-default and import-airframe? Isn't it a runtime default regardless?

dagar · 2020-08-25T15:06:20Z

Needs to be rebased.

dagar · 2020-08-25T15:06:59Z

Alternatively, we could get the core parameter system changes in, then incrementally start transitioning the existing AUTOCNF scripting.

stale · 2020-12-25T12:08:31Z

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

mrpollo · 2021-02-03T16:29:06Z

Feb 3, 2021, Dev Call update: The plan going forward is to split this PR in two, one for the Param architecture, and the second for the parameter per board updates which can happen over time.

mrpollo · 2021-02-10T16:58:12Z

Continued in #16796

RomanBapst requested review from dagar and bkueng March 11, 2020 09:13

RomanBapst added the Admin: Enhancement (improvement) 💡 label Mar 11, 2020

dagar added this to the Release v1.12.0 milestone Mar 12, 2020

dagar reviewed Mar 12, 2020

View reviewed changes

RomanBapst force-pushed the pr_default_params branch from 9162c8b to c359073 Compare March 16, 2020 08:10

RomanBapst added 17 commits March 23, 2020 11:49

parameter system: added support for airframe default parameter

84f7fda

- added param import-as-default command - support storing a default parameter value set at run-time Signed-off-by: RomanBapst <[email protected]>

ROMFS: added airframe parameter files

c5f0025

Signed-off-by: RomanBapst <[email protected]>

board: added board parameter files

af4e8ab

Signed-off-by: RomanBapst <[email protected]>

added vehicle type default parameter files

8730cb6

Signed-off-by: RomanBapst <[email protected]>

added python script which created bison format parameter files from

d2b46b5

human readable parameter files Signed-off-by: RomanBapst <[email protected]>

cmake: generate board-, vehicle_type- and airframe bson parameter files

c8af0f0

Signed-off-by: RomanBapst <[email protected]>

ROMFS: removed autoconfig sections and instead call param import-as-d…

b6e9a65

…efault Signed-off-by: RomanBapst <[email protected]>

added migration scripts

fc6fc5c

ROMFS: removed unused files

8609fd4

Signed-off-by: RomanBapst <[email protected]>

removed unused file

4c1edf1

Signed-off-by: RomanBapst <[email protected]>

do not generate param files for excluded airframes

56c61d7

Signed-off-by: RomanBapst <[email protected]>

avoid using cmake GLOB

b2807cc

Signed-off-by: RomanBapst <[email protected]>

fixed rebase artifact

c85af74

Signed-off-by: RomanBapst <[email protected]>

parameters.cpp: do not story modified parameter if default

7f79d46

Signed-off-by: RomanBapst <[email protected]>

added missing board param file for fmuv5

b201590

Signed-off-by: RomanBapst <[email protected]>

use SYS_AUTOSTART param to determine airframe parameter file

943451c

Signed-off-by: RomanBapst <[email protected]>

ROMFS: added mission airframe parameter file for convergence vtol

1ed2f11

Signed-off-by: RomanBapst <[email protected]>

dagar mentioned this pull request Apr 8, 2020

[RFC] Consider setting default IMU_GYRO_RATEMAX to 400 Hz #14614

Merged

weekly-digest bot mentioned this pull request Apr 19, 2020

Weekly Digest (12 April, 2020 - 19 April, 2020) #14700

Closed

RomanBapst marked this pull request as draft April 28, 2020 07:29

weekly-digest bot mentioned this pull request May 3, 2020

Weekly Digest (26 April, 2020 - 3 May, 2020) #14814

Closed

dagar mentioned this pull request Jun 15, 2020

ROMFS: rc.vtol_defaults limit inner loop rate #13196

Merged

stale bot added the stale label Jul 27, 2020

stale bot removed the stale label Jul 28, 2020

dagar mentioned this pull request Aug 25, 2020

parameters: delete unused PARAM_TYPE_STRUCT #15601

Merged

PX4 deleted a comment from stale bot Aug 25, 2020

dagar self-assigned this Aug 25, 2020

dagar mentioned this pull request Sep 14, 2020

multiple devices for mtd, separate (factory) calibration storage #15731

Merged

stale bot added the stale label Dec 25, 2020

dagar mentioned this pull request Jan 7, 2021

boards: start enabling multi-EKF by default on F7/H7 #16507

Merged

LorenzMeier unassigned dagar Jan 17, 2021

stale bot removed the stale label Jan 17, 2021

LorenzMeier assigned RomanBapst Jan 17, 2021

dagar mentioned this pull request Feb 3, 2021

Discussion: vehicle responsiveness configuration #16744

Merged

dagar mentioned this pull request Feb 9, 2021

parameter runtime defaults and other small improvements #16796

Merged

mrpollo closed this Feb 10, 2021

RomanBapst mentioned this pull request Feb 11, 2021

Parameter System: Added support for airframe default parameter #16819

Closed

dagar deleted the pr_default_params branch January 7, 2022 19:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parameter system enhancement #14363

Parameter system enhancement #14363

RomanBapst commented Mar 11, 2020

dagar commented Mar 12, 2020

dagar commented Mar 12, 2020

dagar Mar 12, 2020

RomanBapst Mar 16, 2020

BazookaJoe1900 commented Mar 12, 2020 •

edited

Loading

hamishwillee commented Mar 12, 2020

RomanBapst commented Mar 16, 2020 •

edited

Loading

RomanBapst commented Mar 16, 2020

RomanBapst commented Mar 16, 2020

bys1123 commented Apr 18, 2020

hamishwillee commented Jul 28, 2020

dagar commented Aug 25, 2020

dagar commented Aug 25, 2020

dagar commented Aug 25, 2020

dagar commented Aug 25, 2020

stale bot commented Dec 25, 2020

mrpollo commented Feb 3, 2021

mrpollo commented Feb 10, 2021

Parameter system enhancement #14363

Parameter system enhancement #14363

Conversation

RomanBapst commented Mar 11, 2020

Short summary

Detailed Summary

Open points for discussion

dagar commented Mar 12, 2020

dagar commented Mar 12, 2020

dagar Mar 12, 2020

Choose a reason for hiding this comment

RomanBapst Mar 16, 2020

Choose a reason for hiding this comment

BazookaJoe1900 commented Mar 12, 2020 • edited Loading

hamishwillee commented Mar 12, 2020

RomanBapst commented Mar 16, 2020 • edited Loading

RomanBapst commented Mar 16, 2020

RomanBapst commented Mar 16, 2020

bys1123 commented Apr 18, 2020

hamishwillee commented Jul 28, 2020

dagar commented Aug 25, 2020

dagar commented Aug 25, 2020

dagar commented Aug 25, 2020

dagar commented Aug 25, 2020

stale bot commented Dec 25, 2020

mrpollo commented Feb 3, 2021

mrpollo commented Feb 10, 2021

BazookaJoe1900 commented Mar 12, 2020 •

edited

Loading

RomanBapst commented Mar 16, 2020 •

edited

Loading