Dynamic detection of host functionality

[alexcrichton]

This is likely a pretty far-future issue, but I was curious to start a conversation around it early on to see what others think!

As wasi continues to grow syscalls (in one way or another) it's inevitable that syscalls will be implemented in some implementations of wasi but not others. Furthermore some host environments may just simply choose to not provide wasi syscall functionality while others do indeed support it. A standard library compiled to wasm, however, needs some sort of policy about what it can and can't do.

For example would a standard library have to specify a minimum set of syscalls it needs to operate? Would it need a specific threshold saying "your implementation must be as new as X with these syscalls"? I ask about this in the context of implementing the Rust standard library for wasi in contrast with how some other platforms are implemented.

Some examples that have come up elsewhere in the past are:

• On Unix we've got syscalls like accept4 which allow atomically setting CLOEXEC, but it's not available on all kernels that the Rust standard library supports. As a result we dynamically detect at runtime whether accept4 is available and fall back to a "guaranteed to work" accept-then-CLOEXEC if it's not found.

• Some platforms support the idea of the "name of thread", primarily used for debugging when you attach gdb for example. We dynamically detect whether linux platforms have the right symbol we can call to configure this, but if it's not present it's just an opportunistic thing that we can safely ignore.

I'd be curious if we could develop something like this for wasi? For example if a syscall was added to aid in debugging (like the thread intrinsic) then the Rust standard library would want to immediately start using but wouldn't want to require all wasi implementations to provide it.

An example strawman API for this might be something like:

int __wasi_syscall_get(const char* module, const char* name, size_t *function_pointer);

where the wasi runtime would effectively attempt to dynamically import name from module (like a static wasm import). If it doens't find anything then the host would return an error, but otherwise a function would be appended to the instance's runtime table of functions and an index to that would be returned through function_pointer. It'd be up to the implementation to then ensure it has the right type signature of the syscall.

I suspect this is also related to #2 in that it could affect what APIs are used, although it's less so about statically requesting different behavior but rather dynamically requesting such behavior.

[npmccallum]

This is precisely why we need modularity and versioning. Perhaps we could add some capability flags as a custom section? Capability flags could specify optional behavior such as CLOEXEC on accept4().

[PoignardAzur]

int __wasi_syscall_get(const char* module, const char* name, size_t *function_pointer);

I think this function might useful in a few corner cases, but most of the time you wouldn't want to use it.

The name, module and type of the function you want to import is usually known as compile-time. What isn't known is whether the function will actually be there.

So what you want is optional imports, which should have three possible outcomes:

• The function doesn't exist in the host environment: No error, the imported value is set to null.

• The function exists and its type matches: No error, the function is imported.

• The function exists, but has an incompatible type: Import error.

[binji]

So what you want is optional imports, which should have three possible outcomes:

This is currently possible, in a not very nice way. You can have the host provide a table of functions, and then import a global i32 which is an index into that table. You can then call the function via call_indirect. If the function is not provided by the host, that table slot will be empty, and will trap when called.

(type $t (func ...))
(import "my" "table" (table ...))
(import "my" "func" (global $g i32))
...
(call_indirect $type (global.get $g) ...)

When typed function references are added, we can do a little better. Typed function references are currently spec'd to be not nullable, so you couldn't import them directly. But assuming we add a downcast from funcref to a typed function reference, you could import that instead. We could also provide the nullable optref $t for this purpose:

(type $t (func ...))
(import "my" "func" (global $g (optref $t)))
...
(call_ref ... (global.get $g))

[PoignardAzur]

Typed function references are currently spec'd to be not nullable

Well that seems like a big oversight.

I get that designers are wary of the "all pointers are nullable" design that was common for a while, but not having optional typed pointers at all is bad too.

[rossberg]

@binji, @PoignardAzur, it's currently listed as an open design question whether the function reference proposal should also include optref (currently left to the GC proposal), i.e., nullable typed references and respective checked conversions. If there is a strong use case for functions then that's certainly possible.

[sbc100]

This would be solved by #36 right? i.e. having a weak import that is runtime testable?

[sunfishcode]

Yes, this seems addressed by #36. If people discover anything which optional imports don't cover, please file new issues!