[WIP] make ast lowering incremental #127262

fee1-dead · 2024-07-03T08:35:11Z

Co-authored-by: Camille Gillot <[email protected]>

This allows to drop the AST when lowering is done.

rustbot · 2024-07-03T08:35:19Z

rustbot has assigned @fmease.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

fee1-dead · 2024-07-03T08:35:32Z

@bors try @rust-timer queue

[WIP] make ast lowering incremental cc `@cjgillot`

bors · 2024-07-03T08:37:52Z

⌛ Trying commit 1a928e6 with merge 82fc609...

fee1-dead · 2024-07-03T08:38:58Z

suppressed some unused warnings to make CI build first

@bors try

[WIP] make ast lowering incremental cc `@cjgillot`

bors · 2024-07-03T08:40:09Z

⌛ Trying commit f04016b with merge c4a0b70...

rust-log-analyzer · 2024-07-03T08:45:38Z

The job mingw-check-tidy failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

Getting action download info
Download action repository 'msys2/[email protected]' (SHA:cc11e9188b693c2b100158c3322424c4cc1dadea)
Download action repository 'actions/checkout@v4' (SHA:692973e3d937129bcbf40652eb9f2f61becf3332)
Download action repository 'actions/upload-artifact@v4' (SHA:65462800fd760344b1a7b4382951275a0abb4808)
Complete job name: PR - mingw-check-tidy
git config --global core.autocrlf false
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
COPY scripts/sccache.sh /scripts/
RUN sh /scripts/sccache.sh

COPY host-x86_64/mingw-check/reuse-requirements.txt /tmp/
RUN pip3 install --no-deps --no-cache-dir --require-hashes -r /tmp/reuse-requirements.txt \
    && pip3 install virtualenv
COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
COPY host-x86_64/mingw-check/validate-error-codes.sh /scripts/

# NOTE: intentionally uses python2 for x.py so we can test it still works.
# NOTE: intentionally uses python2 for x.py so we can test it still works.
# validate-toolstate only runs in our CI, so it's ok for it to only support python3.
ENV SCRIPT TIDY_PRINT_DIFF=1 python2.7 ../x.py test \
           --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes reuse-requirements.in
---

#12 [5/8] COPY host-x86_64/mingw-check/reuse-requirements.txt /tmp/
#12 DONE 0.0s

#13 [6/8] RUN pip3 install --no-deps --no-cache-dir --require-hashes -r /tmp/reuse-requirements.txt     && pip3 install virtualenv
#13 0.428   Downloading binaryornot-0.4.4-py2.py3-none-any.whl (9.0 kB)
#13 0.444 Collecting boolean-py==4.0
#13 0.451   Downloading boolean.py-4.0-py3-none-any.whl (25 kB)
#13 0.467 Collecting chardet==5.1.0
---
#13 3.836 Building wheels for collected packages: reuse
#13 3.837   Building wheel for reuse (pyproject.toml): started
#13 4.157   Building wheel for reuse (pyproject.toml): finished with status 'done'
#13 4.158   Created wheel for reuse: filename=reuse-1.1.0-cp310-cp310-manylinux_2_35_x86_64.whl size=181117 sha256=f5f58750481f69515c2c0d1d503daf565e2565c370d07fc6aeb95fe3498b4269
#13 4.158   Stored in directory: /tmp/pip-ephem-wheel-cache-2bnufuvq/wheels/c2/3c/b9/1120c2ab4bd82694f7e6f0537dc5b9a085c13e2c69a8d0c76d
#13 4.161 Installing collected packages: boolean-py, binaryornot, setuptools, reuse, python-debian, markupsafe, license-expression, jinja2, chardet
#13 4.182   Attempting uninstall: setuptools
#13 4.183     Found existing installation: setuptools 59.6.0
#13 4.184     Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
---
#13 5.413   Downloading virtualenv-20.26.3-py3-none-any.whl (5.7 MB)
#13 5.615      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 28.5 MB/s eta 0:00:00
#13 5.675 Collecting filelock<4,>=3.12.2
#13 5.682   Downloading filelock-3.15.4-py3-none-any.whl (16 kB)
#13 5.702 Collecting distlib<1,>=0.3.7
#13 5.709   Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB)
#13 5.719      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.9/468.9 KB 62.8 MB/s eta 0:00:00
#13 5.750 Collecting platformdirs<5,>=3.9.1
#13 5.757   Downloading platformdirs-4.2.2-py3-none-any.whl (18 kB)
#13 5.847 Installing collected packages: distlib, platformdirs, filelock, virtualenv
#13 6.018 Successfully installed distlib-0.3.8 filelock-3.15.4 platformdirs-4.2.2 virtualenv-20.26.3
#13 DONE 6.1s

#14 [7/8] COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
#14 DONE 0.0s
---
DirectMap4k:      192448 kB
DirectMap2M:     7147520 kB
DirectMap1G:    11534336 kB
##[endgroup]
Executing TIDY_PRINT_DIFF=1 python2.7 ../x.py test            --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
+ TIDY_PRINT_DIFF=1 python2.7 ../x.py test --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
    Finished `dev` profile [unoptimized] target(s) in 0.03s
##[endgroup]
downloading https://ci-artifacts.rust-lang.org/rustc-builds-alt/d68fe4eaa81305b53cdddc2339e8efdc2aa1e013/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
extracting /checkout/obj/build/cache/llvm-d68fe4eaa81305b53cdddc2339e8efdc2aa1e013-true/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz to /checkout/obj/build/x86_64-unknown-linux-gnu/ci-llvm
---
fmt check
fmt: checked 5193 files
tidy check
tidy: Skipping binary file check, read-only filesystem
##[error]tidy error: /checkout/compiler/rustc_ast_lowering/src/delegation.rs:256: TODO is used for tasks that should be done before merging a PR; If you want to leave a message in the codebase use FIXME
##[error]tidy error: /checkout/compiler/rustc_ast_lowering/src/delegation.rs:346: TODO is used for tasks that should be done before merging a PR; If you want to leave a message in the codebase use FIXME
removing old virtual environment
creating virtual environment at '/checkout/obj/build/venv' using 'python3.10'
Requirement already satisfied: pip in ./build/venv/lib/python3.10/site-packages (24.1)
  Downloading pip-24.1.1-py3-none-any.whl.metadata (3.6 kB)
Downloading pip-24.1.1-py3-none-any.whl (1.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 50.6 MB/s eta 0:00:00
Installing collected packages: pip

rust-log-analyzer · 2024-07-03T08:49:10Z

The job dist-x86_64-linux failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

file:.git/config remote.origin.url=https://github.com/rust-lang-ci/rust
file:.git/config remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config gc.auto=0
file:.git/config http.https://github.com/.extraheader=AUTHORIZATION: basic ***
file:.git/config branch.try.remote=origin
file:.git/config branch.try.merge=refs/heads/try
file:.git/config submodule.library/backtrace.url=https://github.com/rust-lang/backtrace-rs.git
file:.git/config submodule.library/stdarch.active=true
file:.git/config submodule.library/stdarch.url=https://github.com/rust-lang/stdarch.git
file:.git/config submodule.src/doc/book.active=true
---
[RUSTC-TIMING] rustc_llvm test:false 0.151
error: unused variable: `param`
    --> compiler/rustc_ast_lowering/src/lib.rs:1694:48
     |
1694 |                 LifetimeRes::Fresh { id, kind, param, .. } => {
     |                                                |
     |                                                help: try removing the field
     |
     = note: `-D unused-variables` implied by `-D warnings`
---
Caused by:
    Command RUST_BACKTRACE=full python3 /checkout/x.py build --target x86_64-unknown-linux-gnu --host x86_64-unknown-linux-gnu --stage 2 library/std --rust-profile-generate /tmp/tmp-multistage/opt-artifacts/rustc-pgo --set llvm.thin-lto=false --set llvm.link-shared=true [at /checkout/obj] has failed with exit code Some(1)

Stack backtrace:
   0: <anyhow::Error>::msg::<alloc::string::String>
             at /rust/deps/anyhow-1.0.86/src/backtrace.rs:27:14
   1: <opt_dist::exec::CmdBuilder>::run
             at /rustc/82fc6097b486101911f5297fa16d6eb9d3e52786/src/tools/opt-dist/src/exec.rs:78:17
   2: <opt_dist::exec::Bootstrap>::run
             at /rustc/82fc6097b486101911f5297fa16d6eb9d3e52786/src/tools/opt-dist/src/exec.rs:179:9
             at /rustc/82fc6097b486101911f5297fa16d6eb9d3e52786/src/tools/opt-dist/src/main.rs:225:13
             at /rustc/82fc6097b486101911f5297fa16d6eb9d3e52786/src/tools/opt-dist/src/main.rs:225:13
   4: <opt_dist::timer::TimerSection>::section::<opt_dist::execute_pipeline::{closure#1}::{closure#0}, ()>
             at /rustc/82fc6097b486101911f5297fa16d6eb9d3e52786/src/tools/opt-dist/src/timer.rs:111:22
             at /rustc/82fc6097b486101911f5297fa16d6eb9d3e52786/src/tools/opt-dist/src/main.rs:214:9
             at /rustc/82fc6097b486101911f5297fa16d6eb9d3e52786/src/tools/opt-dist/src/main.rs:214:9
   6: <opt_dist::timer::TimerSection>::section::<opt_dist::execute_pipeline::{closure#1}, opt_dist::training::RustcPGOProfile>
             at /rustc/82fc6097b486101911f5297fa16d6eb9d3e52786/src/tools/opt-dist/src/timer.rs:111:22
             at /rustc/82fc6097b486101911f5297fa16d6eb9d3e52786/src/tools/opt-dist/src/main.rs:211:29
   8: opt_dist::main
             at /rustc/82fc6097b486101911f5297fa16d6eb9d3e52786/src/tools/opt-dist/src/main.rs:401:18
   9: <fn() -> core::result::Result<(), anyhow::Error> as core::ops::function::FnOnce<()>>::call_once
   9: <fn() -> core::result::Result<(), anyhow::Error> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/75ac3b6331873133c4f7a10f2252afd6f3906c6a/library/core/src/ops/function.rs:250:5
  10: std::sys_common::backtrace::__rust_begin_short_backtrace::<fn() -> core::result::Result<(), anyhow::Error>, core::result::Result<(), anyhow::Error>>
             at /rustc/75ac3b6331873133c4f7a10f2252afd6f3906c6a/library/std/src/sys_common/backtrace.rs:155:18
  11: std::rt::lang_start::<core::result::Result<(), anyhow::Error>>::{closure#0}
             at /rustc/75ac3b6331873133c4f7a10f2252afd6f3906c6a/library/std/src/rt.rs:159:18
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
  13: std::panicking::try::do_call
             at /rustc/75ac3b6331873133c4f7a10f2252afd6f3906c6a/library/std/src/panicking.rs:559:40
  14: std::panicking::try
             at /rustc/75ac3b6331873133c4f7a10f2252afd6f3906c6a/library/std/src/panicking.rs:523:19

bors · 2024-07-03T10:24:17Z

☀️ Try build successful - checks-actions
Build commit: c4a0b70 (c4a0b7041d20e83220aecda15c4efabd6fa1595d)

rust-timer · 2024-07-03T14:48:41Z

Finished benchmarking commit (c4a0b70): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.2%, 2.0%]	126
Regressions ❌ (secondary)	1.0%	[0.2%, 3.8%]	55
Improvements ✅ (primary)	-0.3%	[-0.5%, -0.3%]	3
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.7%	[-0.5%, 2.0%]	129

Max RSS (memory usage)

Results (primary 4.9%, secondary 6.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	4.9%	[1.6%, 22.4%]	112
Regressions ❌ (secondary)	6.6%	[2.7%, 15.5%]	14
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.6%	[-3.6%, -3.6%]	1
All ❌✅ (primary)	4.9%	[1.6%, 22.4%]	112

Cycles

Results (primary 2.0%, secondary 3.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.0%	[0.6%, 3.5%]	65
Regressions ❌ (secondary)	3.7%	[1.3%, 5.6%]	25
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.0%	[0.6%, 3.5%]	65

Binary size

Results (primary 0.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.0%, 0.4%]	26
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.0%	[-0.1%, -0.0%]	17
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.0%	[-0.1%, 0.4%]	43

Bootstrap: 696.734s -> 697.888s (0.17%)
Artifact size: 327.74 MiB -> 327.78 MiB (0.01%)

JohnCSimon · 2024-09-03T22:15:25Z

@fee1-dead
ping from triage - can you post your status on this PR? This PR has not received an update in a few months.

fee1-dead · 2024-09-06T14:40:07Z

Experimental. Not expecting next steps.

Make lowering incremental, take 3/N Rebase of #88186 that was rebased by #127262 No real change in design, so I'm not expecting a perf miracle. r? `@ghost`

fee1-dead and others added 6 commits July 1, 2024 11:34

Do not modify resolver outputs during lowering

99424c3

Co-authored-by: Camille Gillot <[email protected]>

Make lowering incremental.

269d5f1

Co-authored-by: Camille Gillot <[email protected]>

Use some kind of owning_ref for index_ast.

921609a

This allows to drop the AST when lowering is done.

Move HIR traversal queries in rustc_ast_lowering.

7de5d75

Remove hir_crate query.

02982fa

Implement anonymous lifetime lowering as a query

1a928e6

rustbot assigned fmease Jul 3, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels Jul 3, 2024

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 3, 2024

bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 3, 2024

Auto merge of rust-lang#127262 - fee1-dead-contrib:lower-incr2, r=<try>

82fc609

[WIP] make ast lowering incremental cc `@cjgillot`

[]WIP] make CI pass by supressing warnings temporarily

f04016b

petrochenkov self-assigned this Jul 3, 2024

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 3, 2024

bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 3, 2024

Auto merge of rust-lang#127262 - fee1-dead-contrib:lower-incr2, r=<try>

c4a0b70

[WIP] make ast lowering incremental cc `@cjgillot`

petrochenkov removed their assignment Jul 3, 2024

This comment has been minimized.

Sign in to view

fmease assigned cjgillot and unassigned fmease Jul 3, 2024

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Jul 3, 2024

fee1-dead closed this Sep 6, 2024

cjgillot mentioned this pull request Jun 21, 2025

Make lowering incremental, take 3/N #142830

Draft

bors added a commit that referenced this pull request Jun 21, 2025

Auto merge of #142830 - cjgillot:lower-incr-2, r=<try>

a6b131d

Make lowering incremental, take 3/N Rebase of #88186 that was rebased by #127262 No real change in design, so I'm not expecting a perf miracle. r? `@ghost`

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] make ast lowering incremental #127262

[WIP] make ast lowering incremental #127262

Uh oh!

fee1-dead commented Jul 3, 2024

Uh oh!

rustbot commented Jul 3, 2024

Uh oh!

fee1-dead commented Jul 3, 2024

Uh oh!

This comment has been minimized.

bors commented Jul 3, 2024

Uh oh!

fee1-dead commented Jul 3, 2024

Uh oh!

bors commented Jul 3, 2024

Uh oh!

rust-log-analyzer commented Jul 3, 2024

Uh oh!

rust-log-analyzer commented Jul 3, 2024

Uh oh!

bors commented Jul 3, 2024

Uh oh!

This comment has been minimized.

rust-timer commented Jul 3, 2024

Uh oh!

JohnCSimon commented Sep 3, 2024

Uh oh!

fee1-dead commented Sep 6, 2024

Uh oh!

Uh oh!

[WIP] make ast lowering incremental #127262

[WIP] make ast lowering incremental #127262

Uh oh!

Conversation

fee1-dead commented Jul 3, 2024

Uh oh!

rustbot commented Jul 3, 2024

Uh oh!

fee1-dead commented Jul 3, 2024

Uh oh!

This comment has been minimized.

bors commented Jul 3, 2024

Uh oh!

fee1-dead commented Jul 3, 2024

Uh oh!

bors commented Jul 3, 2024

Uh oh!

rust-log-analyzer commented Jul 3, 2024

Uh oh!

rust-log-analyzer commented Jul 3, 2024

Uh oh!

bors commented Jul 3, 2024

Uh oh!

This comment has been minimized.

rust-timer commented Jul 3, 2024

Overall result: ❌ regressions - ACTION NEEDED

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

JohnCSimon commented Sep 3, 2024

Uh oh!

fee1-dead commented Sep 6, 2024

Uh oh!

Uh oh!