Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make Django image use Python 3.10 #9638

Open
MarkusH opened this issue Feb 8, 2023 · 1 comment
Open

Make Django image use Python 3.10 #9638

MarkusH opened this issue Feb 8, 2023 · 1 comment

Comments

@MarkusH
Copy link
Contributor

MarkusH commented Feb 8, 2023 •
edited
Loading

Django 5.0 will require Python 3.10 or newer. The current Python base image seems to only have Python 3.8 available.

Starting Step #3 - "compile-libfuzzer-address-x86_64"
Step #3 - "compile-libfuzzer-address-x86_64": Already have image (with digest): gcr.io/cloud-builders/docker
Step #3 - "compile-libfuzzer-address-x86_64": ---------------------------------------------------------------
Step #3 - "compile-libfuzzer-address-x86_64": Compiling libFuzzer to /usr/lib/libFuzzingEngine.a...  done.
Step #3 - "compile-libfuzzer-address-x86_64": ---------------------------------------------------------------
Step #3 - "compile-libfuzzer-address-x86_64": CC=clang
Step #3 - "compile-libfuzzer-address-x86_64": CXX=clang++
Step #3 - "compile-libfuzzer-address-x86_64": CFLAGS=-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -fno-sanitize=function,leak,vptr,
Step #3 - "compile-libfuzzer-address-x86_64": CXXFLAGS=-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ -fno-sanitize=function,leak,vptr
Step #3 - "compile-libfuzzer-address-x86_64": RUSTFLAGS=--cfg fuzzing -Zsanitizer=address -Cdebuginfo=1 -Cforce-frame-pointers
Step #3 - "compile-libfuzzer-address-x86_64": ---------------------------------------------------------------
Step #3 - "compile-libfuzzer-address-x86_64": + pip3 install .
Step #3 - "compile-libfuzzer-address-x86_64": Processing /src/django
Step #3 - "compile-libfuzzer-address-x86_64":   Installing build dependencies ... done
Step #3 - "compile-libfuzzer-address-x86_64":   Getting requirements to build wheel ... done
Step #3 - "compile-libfuzzer-address-x86_64":   Installing backend dependencies ... done
Step #3 - "compile-libfuzzer-address-x86_64":     Preparing wheel metadata ... done
Step #3 - "compile-libfuzzer-address-x86_64": ERROR: Package 'Django-5.0.dev20230207193000' requires a different Python: 3.8.3 not in '>=3.10'
Step #3 - "compile-libfuzzer-address-x86_64": WARNING: You are using pip version 19.2.3, however version 23.0 is available.
Step #3 - "compile-libfuzzer-address-x86_64": You should consider upgrading via the 'pip install --upgrade pip' command.
Step #3 - "compile-libfuzzer-address-x86_64": ********************************************************************************
Step #3 - "compile-libfuzzer-address-x86_64": Failed to build.
Step #3 - "compile-libfuzzer-address-x86_64": To reproduce, run:
Step #3 - "compile-libfuzzer-address-x86_64": python infra/helper.py build_image django
Step #3 - "compile-libfuzzer-address-x86_64": python infra/helper.py build_fuzzers --sanitizer address --engine libfuzzer --architecture x86_64 django
Step #3 - "compile-libfuzzer-address-x86_64": ********************************************************************************
Finished Step #3 - "compile-libfuzzer-address-x86_64"
ERROR
ERROR: build step 3 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1
MarkusH added a commit to MarkusH/oss-fuzz that referenced this issue Feb 8, 2023
@MarkusH
[Django] Install Python 3.10
8054bff 
Refs google#9638
MarkusH added a commit to MarkusH/oss-fuzz that referenced this issue Feb 8, 2023
@MarkusH
Update build.sh
8cf009a 
Fixes google#9638
DaveLak added a commit to DaveLak/oss-fuzz that referenced this issue Jun 4, 2024
@DaveLak
[django] Fix Coverage Build by Using Base Runner Provided Python
c724d61 
Replaces the ad-hoc workaround using the Dead Snakes PPA to install
Python 3.10, with the upgraded Python 3.10 version provided by
the base-builder and base-runner images, the latter of which resolves
the issue mentioned in google#11419.
Fixes: google#9638
@DaveLak DaveLak mentioned this issue Jun 4, 2024
Merged
@hugovk hugovk mentioned this issue Jul 16, 2024
Merged
DaveLak added a commit to DaveLak/oss-fuzz that referenced this issue Aug 28, 2024
@DaveLak
[django] Fix Coverage Build by Using Base Runner Provided Python
185ac6b 
Replaces the ad-hoc workaround using the Dead Snakes PPA to install
Python 3.10, with the upgraded Python 3.10 version provided by
the base-builder and base-runner images, the latter of which resolves
the issue mentioned in google#11419.
Fixes: google#9638
DaveLak added a commit to DaveLak/oss-fuzz that referenced this issue Nov 20, 2024
@DaveLak
[django] Fix Coverage Build by Using Base Runner Provided Python
8446d53 
Replaces the ad-hoc workaround using the Dead Snakes PPA to install
Python 3.10, with the upgraded Python 3.10 version provided by
the base-builder and base-runner images, the latter of which resolves
the issue mentioned in google#11419.
Fixes: google#9638
oliverchang added a commit that referenced this issue Nov 25, 2024
@DaveLak @oliverchang @radarhere
[infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images (
93b417e 
#12027)

> [!NOTE]  
> I was looking for somewhere to get feedback from maintainers about
this approach to the Python 3.10 upgrade before attempting it, but the
discussion surrounding a Python upgrade has been rather fragmented
across many issues, PRs, and comment chains.
>
> For that reason, I felt it would be easier to propose with a working
example and dedicated PR.


#### Fixes:
- #11419
- #9638

#### Supersedes:
- #9532
- #11420


## Changes

The changes introduced here upgrade Python from 3.8 to 3.10.14 inside
the base-builder and base-runner images.

### Base Image Changes

| Image | Before Changes | After Changes |

|----------------|------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **base-builder** | Compiled Python 3.8 from source using official
release servers at https://www.python.org/ftp/python/. | Compiles Python
3.10.14 (the latest 3.10 release) from source using official release
servers at https://www.python.org/ftp/python/. |
| **base-runner** | Installed Python 3.8 from the default apt repository
provided by the Ubuntu 20.04 image. | Uses a multi-stage build to copy
the Python 3.10.14 interpreter compiled by the base-builder image,
ensuring version sync and saving build time by re-using a pre-built
version. |


## Known Impact on Projects

### 3.9 Workarounds That Can Be Removed

| Project    | Fix Link |
|------------|----------|
| dask |
DaveLak@417bbf5
|
| docutils |
DaveLak@e4c21ff
|
| dovecot |
DaveLak@7ab3ab6
|
| nbclassic |
DaveLak@5509b4e
|
| pandas |
DaveLak@0642a7a
|
| pybind11 |
DaveLak@a5bbdb3
|
| pyodbc |
DaveLak@afa2b5e
|
| qpid-proton|
DaveLak@f5bf756
|

### Anticipated Build Failures

#### Preexisting Failures 

##### Fix is Prepared

| Project               | Fix Link |
|-----------------------|----------|
| airflow |
DaveLak@60a0368
|
| ipython |
DaveLak@21ac68e
|
| networkx |
DaveLak@fc2f8c5
|
| numpy |
DaveLak@9383c87
|
| tensorflow-addons |
DaveLak@eed2bea
|
| django (coverage build)|
DaveLak@c724d61
|
| proto-plus-python |
DaveLak@37d973e
|
| dnspython | The upgraded pip version in the base-builder fixes the
currently failing build. |

##### Fix Requires Upstream Changes

| Project | Issue |
|---------|-------|
| pyvex | Currently failing on python 3.9 because `archinfo` dependency
requires >=3.10. Fails after the 3.10 upgrade because [the upstream
build script needs `python3.9` replaced with
`python3`](https://github.com/angr/pyvex/blob/f94c95636a3800c5bbd781ecf1e3fb0c0d9feec4/fuzzing/build.sh#L19-L23).
|

##### Requires More Investigation

| Project            | Issue |
|--------------------|-------|
| matplotlib | Upgrading Python & Pyinstaller does resolve the build
issues, but an error in the fuzz harness is exposed and must be resolved
for check_build to pass. The exception: `TypeError: Parser.non_math()
takes 2 positional arguments but 4 were given" in "File "fuzz_plt.py",
line 43, in TestOneInput`. |
| scipy | Upgrading Python & Pyinstaller does resolve the build issues,
but an error in the build step causes the build to fail. The error seems
related to the linking: "/usr/bin/ld: /usr/bin/ld: DWARF error: invalid
or unhandled FORM value: 0x25". When `export LDFLAGS="-fuse-ld=lld"` is
set, the error becomes: "`ld.lld: error: undefined symbol:
__asan_report_store4`". |
| pandas (Introspector only)| [This workaround in `build.sh` is the
issue](https://github.com/google/oss-fuzz/blob/1515519a665756d8a50a6c46abac8b431e5462ef/projects/pandas/build.sh#L22-L32).
|
| pycrypto | Failing with error: "`SystemError: PY_SSIZE_T_CLEAN macro
must be defined for '#' formats`". Seems like the issue described
[here](https://stackoverflow.com/a/71019907). Pycrypto is deprecated and
this is unlikely to be fixed upstream. |


## Possible Future Improvements

Using the base-builder image in a multi-stage build to copy the pre-
compiled Python into base-runner is effective, but feels like a
workaround that may be introducing tech debt. A cleaner approach would
be to extract the Python compilation into a discrete base image similar
to how `base-clang` works, and use that as the multi-stage builder in
images that need it.

### Fuzz Introspector Caveat

Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is
not expected to introduce any new issues, it was not tested on these
changes and may require additional work.

---

## Motivation

- Python [3.8 is reaching end of life in October
2024](https://devguide.python.org/versions/).
- The [Scientific Python Community already encourages dropping 3.8
support](https://scientific-python.org/specs/spec-0000/).
- This is evident when looking at which projects have resorted to
upgrading to newer Pythons using ad-hoc workarounds (see `numpy`,
`scipy`, `pandas`, etc.)
- It is likely that more Python projects will begin dropping support for
3.8, further increasing the number of broken builds and ad-hoc
workarounds.
- Code coverage does not work on Python projects that use Python 3.10+
syntax.
- Previous attempts at upgrading Python have stalled (see
google/clusterfuzz#3290 (comment)
& the issues linked under "Supersedes" above.)
- In recognition of the fact that OSS-Fuzz maintainers are stretched
thin, I thought I'd give it a shot.

---------

Co-authored-by: Oliver Chang <[email protected]>
Co-authored-by: Andrew Murray <[email protected]>
@radarhere
Copy link
Contributor

#12027 has now been merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@MarkusH @radarhere