Test suggestions in diagnostics with compiletest

[oli-obk]

Currently suggestions are tested either via compile-fail // SUGGESTION foobar comments or in ui tests by looking at the json or the command line output.

The validity of these suggestions requires a human to interpret them. I suggest (pun intended) that we generate a .suggestion and a .suggestion.stderr file together with the .stderr and the .stdout files. The file will only exist if there are any suggestions produced. The .suggestion file will contain the source with all suggestions applied. The .suggestion.stderr file will contain the result from compiling the .suggestion file.

Open questions:

• What to do if a suggestion has multiple replacements that can apply?
• The results might differ depending on the order of applying multiple suggestions
• The ability to disable suggestion testing on specific files, because it'd just be noise

cc @llogiq @Manishearth @mcarton

[oli-obk]

Now that the ui tests have an intermediate json phase, we can extract the relevant info and reuse parts of rustfix in compiletest. Cc @killercup

[killercup]

I've been working towards something like this in https://github.com/killercup/rustfix/pull/44. My hope is to use rustfix as a library in compiletest.

[phansch]

I think this can be closed now that compiletest has rustfix integrated?