Possible to generate arbitrary json objects?

[MTRNord]

Hi I am using this in a rather unusual way currently to fuzz via HTTP.

I am sending json and one field can be a json object with basically any key variation.

So I am wondering what is the smartest way to do this. Is a grammar based mutator the best way to do this?

[teymour-aldridge]

As this is a context-free grammar, I think so. The other way I know of is to use a recursive mutator to generate an enum (most likely with the same structure as serde_json::Value) and then write a function to serialize it to a string (e.g. as https://github.com/bailion/compiler/blob/main/tests/generator/src/lib.rs does, but this becomes messy; JSON does have a much simpler grammar than that example).

There is a chapter on grammar-based mutators in the guide.

[teymour-aldridge]

Just checked. coverage.py is already what they use for the e2e tests. So thats good news. :) So it will work. Just the magic of hooking it all together will be the step thats likely more complex

I have a scheme I think will work, but won't have time to try and implement it before the weekend.

[loiclec]

It is possible indeed to create a custom Sensor/Pool combo that guides the fuzzer using code coverage collected from somewhere else. Although I'm a bit worried about the performance of it. As always, don't hesitate to contact me for help :-)
Note also that it doesn't have to be guided by code coverage, it could be some other kind of feedback as well.

[teymour-aldridge]

My plan was to use Python's introspection to rewrite the synapse programme at runtime to provide coverage data to Fuzzcheck after every request, and then have a Fuzzcheck sensor and pool to interpret this data.

edit: I think the easiest method is to spin up a new CPython instance for each fuzz test, to start recording the coverage data at the start, and then at the end to extract the coverage data and send it to Fuzzcheck.

[MTRNord]

Although I'm a bit worried about the performance of it.

I am kinda at the point where it doesnt really matter anyway as I can only do like 15Hz of requests anyway to the server as I am limited by the speed of the process. (It is running in a single process mode. I could add workers to it, but it just complicates stuff a lot). The things I am hitting are already at a complexity where the computation of the request tends to need 300-800ms depending on what exactly is sent. (Matrix is a distributed/federating chat protocol, so it has to do some rather complex calculations for that. Though arguably and based on data, most of this is just python being slow. There are go and rust implementations which are much faster but compared not feature complete)

[teymour-aldridge]

You might find the --max-complexity <a number> flag helpful, amongst the other suggestions here - lower down in the issue, not the part about the now fixed bug