Is it sound to check whether the bytes of an `Option<&T>` are zero?

[joshlf]

Co-authored with @jswrenn.

In zerocopy, we have a situation where we have a *const Option<&T>. We know that the referent bytes are "as initialized" as the bytes of an Option<&T>, but not necessarily that they are a bit-valid Option<&T>. By "as initialized", we mean that one of the two is true:

• The referent is a bit-valid Option::<&T>::None
• The referent's discriminant represents Some, and its bytes are initialized wherever &T's bytes are initialized

What we need to do is check whether the referent contains all zeroed bytes. If it does, we can soundly treat those bytes as containing a Option::<&T>::None thanks to the NPO (which guarantees the layout of this specific value).

Our problem is this: We're not sure whether it's sound to look at all of the bytes (in other words, to transmute from Option<&T> to [u8; size_of::<Option<&T>>()]) in order to check that they're all zero. Another option we considered was round-tripping via Option<NonNull<T>> and then using NonNull::addr to extract the address.

Any guidance on whether transmuting Option<&T> to either [u8; size_of::<Option<&T>>()] or to Option<NonNull<T>> are sound?

[Lokathor]

If it's definitely either None or Some then it's definitely fully initialized because that is the point of Rust enums.

I'm unclear why you'd do this transmute instead of matching on the value directly or using is_none

[jswrenn]

I'm unclear why you'd do this transmute instead of matching on the value directly or using is_none

Our starting point is a glorified *const Option<&T>. Glorified in that:

• We know it adheres to shared aliasing rules.
• We know that it's non-null.
• We know that the referent is "as initialized" as Option<&T>

...but we don't know the referent is a validly initialized Option<&T>, nor do we know that the referent is validly aligned. We want to check whether the referent is all-zeros, but given these gaps we can't immediately call is_none.

^{If you're curious, here's our current stab at a proof, which involves several stages of somewhat-justified hoop-jumping in order to call is_none (as you suggest).}

[joshlf]

If it's definitely either None or Some then it's definitely fully initialized because that is the point of Rust enums.

I agree, except that IIUC there's one extra step required: all bytes of a &T must always be initialized. In practice I'm sure this is true, but is it guaranteed? The reason I'm skeptical is a) this aspect of the layout isn't documented anywhere and, b) @RalfJung has previously mentioned that pointer-to-int conversions might be UB. If they're UB, then that implies that doing &T -> [u8; N] might also be UB since another of writing that conversion (where t: &T) is ((t as *const T) as usize).to_ne_bytes(), which relies on this maybe-UB conversion.

[RalfJung]

but we don't know the referent is a validly initialized Option<&T>, nor do we know that the referent is validly aligned

Alignment seems to be the key point here? I am not sure which other part of the validity invariant of Option<&T> might be missing. (Well, there is of course the issue around potentially recursive validity of references; not sure if you are referring to that.)

You are right to be cautious with loading this as a usize, as that could indeed be an ptr2int transmute. But what if you load it as *const () instead? IOW:

unsafe fn is_none<T>(ptr: *const Option<&T>) -> bool {
  ptr.cast::<*const ()>().read().is_null()
}

[joshlf]

Ah that hadn't occurred to me! That seems much more clearly reasonable on its surface.

I'm pretty sure this is sound today, but is it guaranteed to always be sound? IIUC, this relies on:

• All bytes of an Option<&T> are initialized
• All byte patterns are valid instances of *const () (looks like this is guaranteed if we interpret "layout" to include the initialized-ness of bytes)

While this isn't a soundness concern, the correctness of this function relies on the fact that there is only one bit representation for Option::<&T>::None. We know thanks to these docs that the all-zeroes pattern is one valid representation for Option::<&T>::None, but that doesn't guarantee that there aren't others. (Again, obviously this is true in practice, but I'm trying to find docs that guarantee it.)

[Lokathor]

So, are you thinking that, perhaps similar to f32::NAN, there could theoretically be more than one bit pattern that's equal to the literal expression None? And thus, if the bits were non-zero they could still be one of the "other" None values?

[RalfJung]

All bytes of an Option<&T> are initialized

Given that all bytes of &T are initialized, and Option<&T> has the same size, I don't see how there could possibly be a padding byte in Option<&T>.

[joshlf]

So, are you thinking that, perhaps similar to f32::NAN, there could theoretically be more than one bit pattern that's equal to the literal expression None? And thus, if the bits were non-zero they could still be one of the "other" None values?

Yeah, exactly. Obviously I don't actually think that'd ever happen, but technically I don't think the docs currently rule it out.

Given that all bytes of &T are initialized

Is your thinking that that's guaranteed by this? (Edit: as far as I can tell, that section only guarantees size and alignment, but nothing about which bytes are initialized.)

[RalfJung]

Is your thinking that that's guaranteed by this?

My thinking is just that this is "obviously" the case, but I don't know what exactly is stably documented where. It is guaranteed by the MiniRust representation relation, but that doesn't help you.

It's hard to be precise in a spec without fully committing to all the details.

[joshlf]

Yeah, that makes sense.

While we're on the subject, maybe you can clear something up for me. It seems inconsistent to say that we can view the bytes of a pointer (ie, &T -> [u8; N]), but we can't do ptr2int (ie, &T -> usize). We know that bytes-to-int is sound ([u8; N] -> usize), so shouldn't we be able to combine that with the first transformation to get sound ptr2int? There seems to be a contradiction here. Am I missing something?

[Lokathor]

My own understanding is that ptr2int transmutes are sound, but they strip provenance and so you can't transmute back to a pointer later and get a usable pointer.

For simply comparing the int to 0 it should be sound to transmute (again, if my understanding is still up to date).

[RalfJung]

It seems inconsistent to say that we can view the bytes of a pointer (ie, &T -> [u8; N]), but we can't do ptr2int (ie, &T -> usize).

Correct. Viewing the bytes of a pointer also does ptr2int transmute and is hence on equally uncharted ground.

The t-opsem working consensus is what @Lokathor said, but so far we haven't felt ready to stably commit to that, and the lang team hasn't blessed this.

[joshlf]

Would it be easy to articulate what degrees of freedom there are in the design space that make this a not-yet-decided question? In other words, what could cause us to decide that ptr2int is UB in itself (rather than merely producing a pointer which is not particularly useful, and on which further operations are likely to be UB)?

In zerocopy, we have a lot of consumers who want to be able to look at the bytes of a pointer, so being able to make progress on this would be great. I'd be happy to do some of the work to move it forward if the gaps are well-known.

[Lokathor]

Well, p as usize is safe code that works on Stable. So to make ptr2int itself be UB we'd have to somehow explain and justify p as usize as being something other than being a ptr2int operation. That basically wouldn't fly.