|
| 1 | +# Adding new tests |
| 2 | + |
| 3 | +**In general, we expect every PR that fixes a bug in rustc to come |
| 4 | +accompanied by a regression test of some kind.** This test should fail |
| 5 | +in master but pass after the PR. These tests are really useful for |
| 6 | +preventing us from repeating the mistakes of the past. |
| 7 | + |
| 8 | +To add a new test, the first thing you generally do is to create a |
| 9 | +file, typically a Rust source file. Test files have a particular |
| 10 | +structure: |
| 11 | + |
| 12 | +- They always begin with the [copyright notice](./conventions.html#copyright); |
| 13 | +- then they should have some kind of |
| 14 | + [comment explaining what the test is about](#explanatory_comment); |
| 15 | +- next, they can have one or more [header commands](#header_commands), which are special |
| 16 | + comments that the test interpreter knows how to interpret. |
| 17 | +- finally, they have the Rust source. This may have various [error |
| 18 | + annotations](#error_annotations) which indicate expected compilation errors or |
| 19 | + warnings. |
| 20 | + |
| 21 | +Depending on the test suite, there may be some other details to be aware of: |
| 22 | + - For [the `ui` test suite](#ui), you need to generate reference output files. |
| 23 | + |
| 24 | +## What kind of test should I add? |
| 25 | + |
| 26 | +It can be difficult to know what kind of test to use. Here are some |
| 27 | +rough heuristics: |
| 28 | + |
| 29 | +- Some tests have specialized needs: |
| 30 | + - need to run gdb or lldb? use the `debuginfo` test suite |
| 31 | + - need to inspect LLVM IR or MIR IR? use the `codegen` or `mir-opt` test suites |
| 32 | + - need to run rustdoc? Prefer a `rustdoc` test |
| 33 | + - need to inspect the resulting binary in some way? Then use `run-make` |
| 34 | +- For most other things, [a `ui` (or `ui-fulldeps`) test](#ui) is to be preferred: |
| 35 | + - `ui` tests subsume both run-pass, compile-fail, and parse-fail tests |
| 36 | + - in the case of warnings or errors, `ui` tests capture the full output, |
| 37 | + which makes it easier to review but also helps prevent "hidden" regressions |
| 38 | + in the output |
| 39 | + |
| 40 | +## Naming your test |
| 41 | + |
| 42 | +We have not traditionally had a lot of structure in the names of |
| 43 | +tests. Moreover, for a long time, the rustc test runner did not |
| 44 | +support subdirectories (it now does), so test suites like |
| 45 | +[`src/test/run-pass`] have a huge mess of files in them. This is not |
| 46 | +considered an ideal setup. |
| 47 | + |
| 48 | +[`src/test/run-pass`]: https://github.com/rust-lang/rust/tree/master/src/test/run-pass/ |
| 49 | + |
| 50 | +For regression tests -- basically, some random snippet of code that |
| 51 | +came in from the internet -- we often just name the test after the |
| 52 | +issue. For example, `src/test/run-pass/issue-12345.rs`. If possible, |
| 53 | +though, it is better if you can put the test into a directory that |
| 54 | +helps identify what piece of code is being tested here (e.g., |
| 55 | +`borrowck/issue-12345.rs` is much better), or perhaps give it a more |
| 56 | +meaningful name. Still, **do include the issue number somewhere**. |
| 57 | + |
| 58 | +When writing a new feature, **create a subdirectory to store your |
| 59 | +tests**. For example, if you are implementing RFC 1234 ("Widgets"), |
| 60 | +then it might make sense to put the tests in directories like: |
| 61 | + |
| 62 | +- `src/test/ui/rfc1234-widgets/` |
| 63 | +- `src/test/run-pass/rfc1234-widgets/` |
| 64 | +- etc |
| 65 | + |
| 66 | +In other cases, there may already be a suitable directory. (The proper |
| 67 | +directory structure to use is actually an area of active debate.) |
| 68 | + |
| 69 | +<a name=explanatory_comment> |
| 70 | + |
| 71 | +## Comment explaining what the test is about |
| 72 | + |
| 73 | +When you create a test file, **include a comment summarizing the point |
| 74 | +of the test immediately after the copyright notice**. This should |
| 75 | +highlight which parts of the test are more important, and what the bug |
| 76 | +was that the test is fixing. Citing an issue number is often very |
| 77 | +helpful. |
| 78 | + |
| 79 | +This comment doesn't have to be super extensive. Just something like |
| 80 | +"Regression test for #18060: match arms were matching in the wrong |
| 81 | +order." might already be enough. |
| 82 | + |
| 83 | +These comments are very useful to others later on when your test |
| 84 | +breaks, since they often can highlight what the problem is. They are |
| 85 | +also useful if for some reason the tests need to be refactored, since |
| 86 | +they let others know which parts of the test were important (often a |
| 87 | +test must be rewritten because it no longer tests what is was meant to |
| 88 | +test, and then it's useful to know what it *was* meant to test |
| 89 | +exactly). |
| 90 | + |
| 91 | +<a name=header_commands> |
| 92 | + |
| 93 | +## Header commands: configuring rustc |
| 94 | + |
| 95 | +Header commands are special comments that the test runner knows how to |
| 96 | +interpret. They must appear before the Rust source in the test. They |
| 97 | +are normally put after the short comment that explains the point of |
| 98 | +this test. For example, this test uses the `// compile-flags` command |
| 99 | +to specify a custom flag to give to rustc when the test is compiled: |
| 100 | + |
| 101 | +```rust |
| 102 | +// Copyright 2017 The Rust Project Developers. blah blah blah. |
| 103 | +// ... |
| 104 | +// except according to those terms. |
| 105 | + |
| 106 | +// Test the behavior of `0 - 1` when overflow checks are disabled. |
| 107 | + |
| 108 | +// compile-flags: -Coverflow-checks=off |
| 109 | + |
| 110 | +fn main() { |
| 111 | + let x = 0 - 1; |
| 112 | + ... |
| 113 | +} |
| 114 | +``` |
| 115 | + |
| 116 | +### Ignoring tests |
| 117 | + |
| 118 | +These are used to ignore the test in some situations, which means the test won't |
| 119 | +be compiled or run. |
| 120 | + |
| 121 | +* `ignore-X` where `X` is a target detail or stage will ignore the test accordingly (see below) |
| 122 | +* `ignore-pretty` will not compile the pretty-printed test (this is done to test the pretty-printer, but might not always work) |
| 123 | +* `ignore-test` always ignores the test |
| 124 | +* `ignore-lldb` and `ignore-gdb` will skip a debuginfo test on that debugger. |
| 125 | + |
| 126 | +Some examples of `X` in `ignore-X`: |
| 127 | + |
| 128 | +* Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`, `x86`, ... |
| 129 | +* OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`, `windows`, ... |
| 130 | +* Environment (fourth word of the target triple): `gnu`, `msvc`, `musl`. |
| 131 | +* Pointer width: `32bit`, `64bit`. |
| 132 | +* Stage: `stage0`, `stage1`, `stage2`. |
| 133 | + |
| 134 | +### Other Header Commands |
| 135 | + |
| 136 | +Here is a list of other header commands. This list is not |
| 137 | +exhaustive. Header commands can generally be found by browsing the |
| 138 | +`TestProps` structure found in [`header.rs`] from the compiletest |
| 139 | +source. |
| 140 | + |
| 141 | +* `min-{gdb,lldb}-version` |
| 142 | +* `min-llvm-version` |
| 143 | +* `must-compile-successfully` for UI tests, indicates that the test is supposed |
| 144 | + to compile, as opposed to the default where the test is supposed to error out. |
| 145 | +* `compile-flags` passes extra command-line args to the compiler, |
| 146 | + e.g. `compile-flags -g` which forces debuginfo to be enabled. |
| 147 | +* `should-fail` indicates that the test should fail; used for "meta testing", |
| 148 | + where we test the compiletest program itself to check that it will generate |
| 149 | + errors in appropriate scenarios. This header is ignored for pretty-printer tests. |
| 150 | +* `gate-test-X` where `X` is a feature marks the test as "gate test" for feature X. |
| 151 | + Such tests are supposed to ensure that the compiler errors when usage of a gated |
| 152 | + feature is attempted without the proper `#![feature(X)]` tag. |
| 153 | + Each unstable lang feature is required to have a gate test. |
| 154 | + |
| 155 | +[`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs |
| 156 | + |
| 157 | +<a name="error_annotations"> |
| 158 | + |
| 159 | +## Error annotations |
| 160 | + |
| 161 | +Error annotations specify the errors that the compiler is expected to |
| 162 | +emit. They are "attached" to the line in source where the error is |
| 163 | +located. |
| 164 | + |
| 165 | +* `~`: Associates the following error level and message with the |
| 166 | + current line |
| 167 | +* `~|`: Associates the following error level and message with the same |
| 168 | + line as the previous comment |
| 169 | +* `~^`: Associates the following error level and message with the |
| 170 | + previous line. Each caret (`^`) that you add adds a line to this, so |
| 171 | + `~^^^^^^^` is seven lines up. |
| 172 | + |
| 173 | +The error levels that you can have are: |
| 174 | + |
| 175 | +1. `ERROR` |
| 176 | +2. `WARNING` |
| 177 | +3. `NOTE` |
| 178 | +4. `HELP` and `SUGGESTION`* |
| 179 | + |
| 180 | +\* **Note**: `SUGGESTION` must follow immediately after `HELP`. |
| 181 | + |
| 182 | +## Revisions |
| 183 | + |
| 184 | +Certain classes of tests support "revisions" (as of the time of this |
| 185 | +writing, this includes run-pass, compile-fail, run-fail, and |
| 186 | +incremental, though incremental tests are somewhat |
| 187 | +different). Revisions allow a single test file to be used for multiple |
| 188 | +tests. This is done by adding a special header at the top of the file: |
| 189 | + |
| 190 | +``` |
| 191 | +// revisions: foo bar baz |
| 192 | +``` |
| 193 | + |
| 194 | +This will result in the test being compiled (and tested) three times, |
| 195 | +once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg |
| 196 | +baz`. You can therefore use `#[cfg(foo)]` etc within the test to tweak |
| 197 | +each of these results. |
| 198 | + |
| 199 | +You can also customize headers and expected error messages to a particular |
| 200 | +revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//` |
| 201 | +comment, like so: |
| 202 | + |
| 203 | +``` |
| 204 | +// A flag to pass in only for cfg `foo`: |
| 205 | +//[foo]compile-flags: -Z verbose |
| 206 | +
|
| 207 | +#[cfg(foo)] |
| 208 | +fn test_foo() { |
| 209 | + let x: usize = 32_u32; //[foo]~ ERROR mismatched types |
| 210 | +} |
| 211 | +``` |
| 212 | + |
| 213 | +Note that not all headers have meaning when customized to a revision. |
| 214 | +For example, the `ignore-test` header (and all "ignore" headers) |
| 215 | +currently only apply to the test as a whole, not to particular |
| 216 | +revisions. The only headers that are intended to really work when |
| 217 | +customized to a revision are error patterns and compiler flags. |
| 218 | + |
| 219 | +<a name="ui"> |
| 220 | + |
| 221 | +## Guide to the UI tests |
| 222 | + |
| 223 | +The UI tests are intended to capture the compiler's complete output, |
| 224 | +so that we can test all aspects of the presentation. They work by |
| 225 | +compiling a file (e.g., [`ui/hello_world/main.rs`][hw-main]), |
| 226 | +capturing the output, and then applying some normalization (see |
| 227 | +below). This normalized result is then compared against reference |
| 228 | +files named `ui/hello_world/main.stderr` and |
| 229 | +`ui/hello_world/main.stdout`. If either of those files doesn't exist, |
| 230 | +the output must be empty (that is actually the case for |
| 231 | +[this particular test][hw]). If the test run fails, we will print out |
| 232 | +the current output, but it is also saved in |
| 233 | +`build/<target-triple>/test/ui/hello_world/main.stdout` (this path is |
| 234 | +printed as part of the test failure message), so you can run `diff` |
| 235 | +and so forth. |
| 236 | + |
| 237 | +[hw-main]: https://github.com/rust-lang/rust/blob/master/src/test/ui/hello_world/main.rs |
| 238 | +[hw]: https://github.com/rust-lang/rust/blob/master/src/test/ui/hello_world/ |
| 239 | + |
| 240 | +### Tests that do not result in compile errors |
| 241 | + |
| 242 | +By default, a UI test is expected **not to compile** (in which case, |
| 243 | +it should contain at least one `//~ ERROR` annotation). However, you |
| 244 | +can also make UI tests where compilation is expected to succeed, and |
| 245 | +you can even run the resulting program. Just add one of the following |
| 246 | +[header commands](#header_commands): |
| 247 | + |
| 248 | +- `// must-compile-successfully` -- compilation should succeed but do not run the resulting binary |
| 249 | +- `// run-pass` -- compilation should succeed and we should run the resulting binary |
| 250 | + |
| 251 | +### Editing and updating the reference files |
| 252 | + |
| 253 | +If you have changed the compiler's output intentionally, or you are |
| 254 | +making a new test, you can use the script `ui/update-references.sh` to |
| 255 | +update the references. When you run the test framework, it will report |
| 256 | +various errors: in those errors is a command you can use to run the |
| 257 | +`ui/update-references.sh` script, which will then copy over the files |
| 258 | +from the build directory and use them as the new reference. You can |
| 259 | +also just run `ui/update-all-references.sh`. In both cases, you can run |
| 260 | +the script with `--help` to get a help message. |
| 261 | + |
| 262 | +### Normalization |
| 263 | + |
| 264 | +The normalization applied is aimed at eliminating output difference |
| 265 | +between platforms, mainly about filenames: |
| 266 | + |
| 267 | +- the test directory is replaced with `$DIR` |
| 268 | +- all backslashes (`\`) are converted to forward slashes (`/`) (for Windows) |
| 269 | +- all CR LF newlines are converted to LF |
| 270 | + |
| 271 | +Sometimes these built-in normalizations are not enough. In such cases, you |
| 272 | +may provide custom normalization rules using the header commands, e.g. |
| 273 | + |
| 274 | +``` |
| 275 | +// normalize-stdout-test: "foo" -> "bar" |
| 276 | +// normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)" |
| 277 | +// normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)" |
| 278 | +``` |
| 279 | + |
| 280 | +This tells the test, on 32-bit platforms, whenever the compiler writes |
| 281 | +`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)` |
| 282 | +instead. Similar for 64-bit. The replacement is performed by regexes using |
| 283 | +default regex flavor provided by `regex` crate. |
| 284 | + |
| 285 | +The corresponding reference file will use the normalized output to test both |
| 286 | +32-bit and 64-bit platforms: |
| 287 | + |
| 288 | +``` |
| 289 | +... |
| 290 | + | |
| 291 | + = note: source type: fn() ($PTR bits) |
| 292 | + = note: target type: u16 (16 bits) |
| 293 | +... |
| 294 | +``` |
| 295 | + |
| 296 | +Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`][] for a concrete usage example. |
| 297 | + |
| 298 | +[mrs]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.rs |
| 299 | +[`main.stderr`]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.stderr |
| 300 | + |
| 301 | +Besides `normalize-stderr-32bit` and `-64bit`, one may use any target |
| 302 | +information or stage supported by `ignore-X` here as well (e.g. |
| 303 | +`normalize-stderr-windows` or simply `normalize-stderr-test` for unconditional |
| 304 | +replacement). |
0 commit comments