test: improve test infrastructure (#554)

This change represents a rather large re-design in how `wasi-libc`
builds and runs its tests. Initially, #346 retrieved the `libc-test`
repository and built a subset of those tests to give us some amount of
test coverage. Later, because there was no way to add custom C tests,
#522 added a `smoke` directory which allowed this. But (a) each of these
test suites was built and run separately and (b) it was unclear how to
add more tests flexibly--some tests should only run on `*p2` targets or
`*-threads` targets, e.g.

This change reworks all of this so that all tests are built the same
way, in the same place. For downloaded tests like those from
`libc-test`, I chose to add "stub tests" that `#include` the original
version. This not only keeps all enabled tests in one place, it also
allows us to add "directives," C comments that the `Makefile` uses to
filter out tests for certain targets or add special compile, link or run
flags. These rudimentary scripts, along with other Bash logic I moved
out of the Makefile now live in the `scripts` directory.

Finally, all of this is explained more clearly in an updated
`README.md`. The hope with documenting this a bit better is that it
would be easier for drive-by contributors to be able to either dump in
new C tests for regressions they may find or enable more libc-tests. As
of my current count, we only enable 40/75 of libc-test's functional
tests, 0/228 math tests, 0/69 regression tests, and 0/79 API tests.
Though many of these may not apply to WASI programs, it would be nice to
explore how many more of these tests can be enabled to increase
wasi-libc's test coverage. This change should explain how to do that
and, with directives, make it possible to condition how the tests
compile and run.

Author: abrown

.github/workflows/main.yml

@@ -125,17 +125,13 @@ jobs:
         export WASIP1_DIR=$(realpath $($CLANG_DIR/clang -print-resource-dir)/lib/wasip1/)
         export WASIP2_DIR=$(realpath $($CLANG_DIR/clang -print-resource-dir)/lib/wasip2/)
         mkdir -p $WASI_DIR $WASIP1_DIR $WASIP2_DIR
-        cp download/libclang_rt.builtins-wasm32.a $WASI_DIR
-        cp download/libclang_rt.builtins-wasm32.a $WASIP1_DIR
-        cp download/libclang_rt.builtins-wasm32.a $WASIP2_DIR
+        cp build/download/libclang_rt.builtins-wasm32.a $WASI_DIR
+        cp build/download/libclang_rt.builtins-wasm32.a $WASIP1_DIR
+        cp build/download/libclang_rt.builtins-wasm32.a $WASIP2_DIR
         TARGET_TRIPLE=wasm32-wasi make test
-        rm -r build
         TARGET_TRIPLE=wasm32-wasip1 make test
-        rm -r build
         TARGET_TRIPLE=wasm32-wasip2 make test
-        rm -r build
         TARGET_TRIPLE=wasm32-wasi-threads make test
-        rm -r build
         TARGET_TRIPLE=wasm32-wasip1-threads make test
       # The older version of Clang does not provide the expected symbol for the
       # test entrypoints: `undefined symbol: __main_argc_argv`.

test/.gitignore

@@ -1,5 +1,2 @@
 build
-download
 run
-
-smoke/*.dir