CIFuzz build failures - _freeze_module segfaults during the build, fuzz testing doesn't start.

[gpshead]

PR authors and core devs have been seeing CIFuzz failures on PRs not related to their changes of late. These are non-blocking, but confusing.

The CIFuzz build step is failing, it seems that _freeze_module crashes:

./Programs/_freeze_module importlib._bootstrap_external ./Lib/importlib/_bootstrap_external.py Python/frozen_modules/importlib._bootstrap_external.h
Segmentation fault (core dumped)

I believe _freeze_module got compiled with memory-sanitizer but no useful MSAN report seems to be generated by that crash.

Unclear where the problem lies, maybe this could be CIFuzz container or clang related?

CIFuzz background: read PR #107653 (discussion and links) and Issue #107652 (the pitch).

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-116886: Temporarily disable CIfuzz (memory) #117018
• Revert "gh-116886: Temporarily disable CIfuzz (memory) (#117018)" #117289

[gpshead]

@illia-v

[freakboy3742]

Looking at CI history, the last PR merged to main that I can find that passed that test was #116761; the first to fail it was #116528. However, the CIFuzz test isn't run on every PR, and PRs aren't necessarily up to date at time of merge, so the trigger for this failure isn't necessarily in that specific merge range - but I'm guessing it will be close.

[freakboy3742]

Another thought - the issue could be with the CI action itself. The CIFuzz job uses the master branch of the actions, so CPython's CI results are going to be dependent on the stability of the upstream repo.

As an experiment, I modified the CIFuzz checkout on an older version of the action, and ran it on a copy of #116454.

If I revert to the 10e6cbd7 commit to oss-fuzz from a couple of days ago, the CIFuzz test fails because of a clang issue failing to compile the code, which is at least a different failure, and demonstrates that there's at least some sensitivity to the oss-fuzz version.

[vstinner]

When I tried to reproduce in an Ubuntu 22.04 container, I get these errors:

root@ubuntu-dev:/cpython# make -j4 SHELL="bash -x"
(...)
./Programs/_freeze_module getpath ./Modules/getpath.py Python/frozen_modules/getpath.h
==11165==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55d06e6084c0 in read_text /cpython/Programs/_freeze_module.c:109:9
    #1 0x55d06e6084c0 in main /cpython/Programs/_freeze_module.c:225:24
    #2 0x7f1097f42d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #3 0x7f1097f42e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #4 0x55d06e580f34 in _start (/cpython/Programs/_freeze_module+0x85f34) (BuildId: f91c6a29210dc0cd457ac4d39f134316a9608c08)

  Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x55d06e606f60 in main /cpython/Programs/_freeze_module.c:206

SUMMARY: MemorySanitizer: use-of-uninitialized-value /cpython/Programs/_freeze_module.c:109:9 in read_text
==11165==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55d06e6084cd in read_text /cpython/Programs/_freeze_module.c:116:21
    #1 0x55d06e6084cd in main /cpython/Programs/_freeze_module.c:225:24
    #2 0x7f1097f42d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #3 0x7f1097f42e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #4 0x55d06e580f34 in _start (/cpython/Programs/_freeze_module+0x85f34) (BuildId: f91c6a29210dc0cd457ac4d39f134316a9608c08)

  Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x55d06e606f60 in main /cpython/Programs/_freeze_module.c:206

SUMMARY: MemorySanitizer: use-of-uninitialized-value /cpython/Programs/_freeze_module.c:116:21 in read_text
MemorySanitizer: 2 warnings reported.
make: *** [Makefile:1640: Python/frozen_modules/getpath.h] Error 1
make: *** Waiting for unfinished jobs....

I can reproduce the issue by running _freeze_module manually:

# ./Programs/_freeze_module zipimport ./Lib/zipimport.py Python/frozen_modules/zipimport.h
==11199==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5622e273f4c0 in read_text /cpython/Programs/_freeze_module.c:109:9
    #1 0x5622e273f4c0 in main /cpython/Programs/_freeze_module.c:225:24
    #2 0x7fa50d770d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #3 0x7fa50d770e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #4 0x5622e26b7f34 in _start (/cpython/Programs/_freeze_module+0x85f34) (BuildId: f91c6a29210dc0cd457ac4d39f134316a9608c08)

  Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x5622e273df60 in main /cpython/Programs/_freeze_module.c:206

SUMMARY: MemorySanitizer: use-of-uninitialized-value /cpython/Programs/_freeze_module.c:109:9 in read_text
==11199==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5622e273f4cd in read_text /cpython/Programs/_freeze_module.c:116:21
    #1 0x5622e273f4cd in main /cpython/Programs/_freeze_module.c:225:24
    #2 0x7fa50d770d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #3 0x7fa50d770e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #4 0x5622e26b7f34 in _start (/cpython/Programs/_freeze_module+0x85f34) (BuildId: f91c6a29210dc0cd457ac4d39f134316a9608c08)

  Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x5622e273df60 in main /cpython/Programs/_freeze_module.c:206

SUMMARY: MemorySanitizer: use-of-uninitialized-value /cpython/Programs/_freeze_module.c:116:21 in read_text
MemorySanitizer: 2 warnings reported.

Commands used to build Python in a fresh Ubuntu 22.04 container:

apt update && apt install --yes git make clang libssl-dev libreadline-dev libc6-dev
git clone https://github.com/python/cpython --depth=1 
cd cpython/
export ASAN_OPTIONS=detect_leaks=0
./configure --with-memory-sanitizer --without-ensurepip 'CFLAGS=-mllvm -msan-keep-going=1' --prefix /opt/python CC=clang LD=clang
make -j4 SHELL="bash -x"

Then just to reproduce the issue:

./Programs/_freeze_module zipimport ./Lib/zipimport.py Python/frozen_modules/zipimport.h

[vstinner]

Oh wait, stat.i is an old issue: #91043

[vstinner]

The stat.i issue can be worked around by:

# git diff
diff --git a/Python/fileutils.c b/Python/fileutils.c
index 882d329957..b1e273d810 100644
--- a/Python/fileutils.c
+++ b/Python/fileutils.c
@@ -1235,6 +1235,7 @@ _Py_stat_basic_info_to_stat(FILE_STAT_BASIC_INFORMATION *info,
 int
 _Py_fstat_noraise(int fd, struct _Py_stat_struct *status)
 {
+    memset(status, 0, sizeof(*status));
 #ifdef MS_WINDOWS
     BY_HANDLE_FILE_INFORMATION info;
     FILE_BASIC_INFO basicInfo;

[illia-v]

I ran CIFuzz for a commit from 3 weeks ago in my fork and it failed whereas the job was succeeding this time ago for PRs continuously.

According to https://oss-fuzz-build-logs.storage.googleapis.com/index.html#cpython3, Google haven't recently had any problems building CPython with the memory sanitizer for OSS-Fuzz.

So, I believe the issue is not with CPython but with the environment created by the action.

[vstinner]

It's annoying that pull request have a red "X" failure sign when "CIFuzz (memory)" is the only failing job, and it's a known failure. Can we disable the job or change GitHub configuration to not mark a whole PR as failed, when a CIFuzz job fails?