[LV] Disable interleaving via hints for uncountable early exit loops #145877
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently if the user enables interleaving during vectorisation of uncountable early exit loops via the interleave_count pragma and the enable-early-exit-vectorization option, it will miscompile. There is ongoing work to fix this, but for now it seems safer to ignore the hint until it is supported.
|
@llvm/pr-subscribers-llvm-transforms Author: David Sherwood (david-arm) ChangesCurrently if the user enables interleaving during vectorisation of uncountable early exit loops via the interleave_count pragma and the enable-early-exit-vectorization option, it will miscompile. There is ongoing work to fix this, but for now it seems safer to ignore the hint until it is supported. Full diff: https://github.com/llvm/llvm-project/pull/145877.diff 2 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index f4259d3d69880..ac65884f341a5 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -10101,6 +10101,13 @@ bool LoopVectorizePass::processLoop(Loop *L) {
// Get user vectorization factor and interleave count.
ElementCount UserVF = Hints.getWidth();
unsigned UserIC = Hints.getInterleave();
+ if (LVL.hasUncountableEarlyExit() && UserIC != 1 &&
+ !VectorizerParams::isInterleaveForced()) {
+ UserIC = 1;
+ reportVectorizationInfo("Interleaving not currently supported for loops "
+ "with uncountable early exits",
+ "InterleaveEarlyExitDisabled", ORE, L);
+ }
// Plan how to best vectorize.
LVP.plan(UserVF, UserIC);
diff --git a/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll b/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll
new file mode 100644
index 0000000000000..c00ce09713279
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll
@@ -0,0 +1,88 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
+; RUN: opt -p loop-vectorize -enable-early-exit-vectorization -force-vector-width=4 \
+; RUN: -debug-only=loop-vectorize -S %s 2>%t | FileCheck --check-prefix=VF4IC4 %s
+; RUN: cat %t | FileCheck --check-prefix=DEBUG %s
+
+declare void @init_mem(ptr, i64);
+
+; DEBUG: Interleaving not currently supported for loops with uncountable early exits
+
+define i64 @multi_exiting_to_different_exits_live_in_exit_values() {
+; VF4IC4-LABEL: define i64 @multi_exiting_to_different_exits_live_in_exit_values() {
+; VF4IC4-NEXT: [[ENTRY:.*]]:
+; VF4IC4-NEXT: [[SRC:%.*]] = alloca [128 x i32], align 4
+; VF4IC4-NEXT: call void @init_mem(ptr [[SRC]])
+; VF4IC4-NEXT: br i1 false, label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; VF4IC4: [[VECTOR_PH]]:
+; VF4IC4-NEXT: br label %[[VECTOR_BODY:.*]]
+; VF4IC4: [[VECTOR_BODY]]:
+; VF4IC4-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; VF4IC4-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[SRC]], i64 [[INDEX]]
+; VF4IC4-NEXT: [[TMP1:%.*]] = getelementptr inbounds i32, ptr [[TMP0]], i32 0
+; VF4IC4-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP1]], align 4
+; VF4IC4-NEXT: [[TMP2:%.*]] = icmp eq <4 x i32> [[WIDE_LOAD]], splat (i32 10)
+; VF4IC4-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; VF4IC4-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
+; VF4IC4-NEXT: [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], 128
+; VF4IC4-NEXT: [[TMP5:%.*]] = or i1 [[TMP3]], [[TMP4]]
+; VF4IC4-NEXT: br i1 [[TMP5]], label %[[MIDDLE_SPLIT:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; VF4IC4: [[MIDDLE_SPLIT]]:
+; VF4IC4-NEXT: br i1 [[TMP3]], label %[[VECTOR_EARLY_EXIT:.*]], label %[[MIDDLE_BLOCK:.*]]
+; VF4IC4: [[MIDDLE_BLOCK]]:
+; VF4IC4-NEXT: br i1 true, label %[[E2:.*]], label %[[SCALAR_PH]]
+; VF4IC4: [[VECTOR_EARLY_EXIT]]:
+; VF4IC4-NEXT: br label %[[E1:.*]]
+; VF4IC4: [[SCALAR_PH]]:
+; VF4IC4-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 128, %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; VF4IC4-NEXT: br label %[[LOOP_HEADER:.*]]
+; VF4IC4: [[LOOP_HEADER]]:
+; VF4IC4-NEXT: [[IV:%.*]] = phi i64 [ [[INC:%.*]], %[[LOOP_LATCH:.*]] ], [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ]
+; VF4IC4-NEXT: [[GEP_SRC:%.*]] = getelementptr inbounds i32, ptr [[SRC]], i64 [[IV]]
+; VF4IC4-NEXT: [[L:%.*]] = load i32, ptr [[GEP_SRC]], align 4
+; VF4IC4-NEXT: [[C_1:%.*]] = icmp eq i32 [[L]], 10
+; VF4IC4-NEXT: br i1 [[C_1]], label %[[E1]], label %[[LOOP_LATCH]]
+; VF4IC4: [[LOOP_LATCH]]:
+; VF4IC4-NEXT: [[INC]] = add nuw i64 [[IV]], 1
+; VF4IC4-NEXT: [[C_2:%.*]] = icmp eq i64 [[INC]], 128
+; VF4IC4-NEXT: br i1 [[C_2]], label %[[E2]], label %[[LOOP_HEADER]], !llvm.loop [[LOOP3:![0-9]+]]
+; VF4IC4: [[E1]]:
+; VF4IC4-NEXT: [[P1:%.*]] = phi i64 [ 0, %[[LOOP_HEADER]] ], [ 0, %[[VECTOR_EARLY_EXIT]] ]
+; VF4IC4-NEXT: ret i64 [[P1]]
+; VF4IC4: [[E2]]:
+; VF4IC4-NEXT: [[P2:%.*]] = phi i64 [ 1, %[[LOOP_LATCH]] ], [ 1, %[[MIDDLE_BLOCK]] ]
+; VF4IC4-NEXT: ret i64 [[P2]]
+;
+entry:
+ %src = alloca [128 x i32]
+ call void @init_mem(ptr %src)
+ br label %loop.header
+
+loop.header:
+ %iv = phi i64 [ %inc, %loop.latch ], [ 0, %entry ]
+ %gep.src = getelementptr inbounds i32, ptr %src, i64 %iv
+ %l = load i32, ptr %gep.src
+ %c.1 = icmp eq i32 %l, 10
+ br i1 %c.1, label %e1, label %loop.latch
+
+loop.latch:
+ %inc = add nuw i64 %iv, 1
+ %c.2 = icmp eq i64 %inc, 128
+ br i1 %c.2, label %e2, label %loop.header, !llvm.loop !0
+
+e1:
+ %p1 = phi i64 [ 0, %loop.header ]
+ ret i64 %p1
+
+e2:
+ %p2 = phi i64 [ 1, %loop.latch ]
+ ret i64 %p2
+}
+
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.interleave.count", i32 4}
+;.
+; VF4IC4: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; VF4IC4: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
+; VF4IC4: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
+; VF4IC4: [[LOOP3]] = distinct !{[[LOOP3]], [[META2]], [[META1]]}
+;.
|
|
@llvm/pr-subscribers-vectorizers Author: David Sherwood (david-arm) ChangesCurrently if the user enables interleaving during vectorisation of uncountable early exit loops via the interleave_count pragma and the enable-early-exit-vectorization option, it will miscompile. There is ongoing work to fix this, but for now it seems safer to ignore the hint until it is supported. Full diff: https://github.com/llvm/llvm-project/pull/145877.diff 2 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index f4259d3d69880..ac65884f341a5 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -10101,6 +10101,13 @@ bool LoopVectorizePass::processLoop(Loop *L) {
// Get user vectorization factor and interleave count.
ElementCount UserVF = Hints.getWidth();
unsigned UserIC = Hints.getInterleave();
+ if (LVL.hasUncountableEarlyExit() && UserIC != 1 &&
+ !VectorizerParams::isInterleaveForced()) {
+ UserIC = 1;
+ reportVectorizationInfo("Interleaving not currently supported for loops "
+ "with uncountable early exits",
+ "InterleaveEarlyExitDisabled", ORE, L);
+ }
// Plan how to best vectorize.
LVP.plan(UserVF, UserIC);
diff --git a/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll b/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll
new file mode 100644
index 0000000000000..c00ce09713279
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll
@@ -0,0 +1,88 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
+; RUN: opt -p loop-vectorize -enable-early-exit-vectorization -force-vector-width=4 \
+; RUN: -debug-only=loop-vectorize -S %s 2>%t | FileCheck --check-prefix=VF4IC4 %s
+; RUN: cat %t | FileCheck --check-prefix=DEBUG %s
+
+declare void @init_mem(ptr, i64);
+
+; DEBUG: Interleaving not currently supported for loops with uncountable early exits
+
+define i64 @multi_exiting_to_different_exits_live_in_exit_values() {
+; VF4IC4-LABEL: define i64 @multi_exiting_to_different_exits_live_in_exit_values() {
+; VF4IC4-NEXT: [[ENTRY:.*]]:
+; VF4IC4-NEXT: [[SRC:%.*]] = alloca [128 x i32], align 4
+; VF4IC4-NEXT: call void @init_mem(ptr [[SRC]])
+; VF4IC4-NEXT: br i1 false, label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; VF4IC4: [[VECTOR_PH]]:
+; VF4IC4-NEXT: br label %[[VECTOR_BODY:.*]]
+; VF4IC4: [[VECTOR_BODY]]:
+; VF4IC4-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; VF4IC4-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[SRC]], i64 [[INDEX]]
+; VF4IC4-NEXT: [[TMP1:%.*]] = getelementptr inbounds i32, ptr [[TMP0]], i32 0
+; VF4IC4-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP1]], align 4
+; VF4IC4-NEXT: [[TMP2:%.*]] = icmp eq <4 x i32> [[WIDE_LOAD]], splat (i32 10)
+; VF4IC4-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; VF4IC4-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
+; VF4IC4-NEXT: [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], 128
+; VF4IC4-NEXT: [[TMP5:%.*]] = or i1 [[TMP3]], [[TMP4]]
+; VF4IC4-NEXT: br i1 [[TMP5]], label %[[MIDDLE_SPLIT:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; VF4IC4: [[MIDDLE_SPLIT]]:
+; VF4IC4-NEXT: br i1 [[TMP3]], label %[[VECTOR_EARLY_EXIT:.*]], label %[[MIDDLE_BLOCK:.*]]
+; VF4IC4: [[MIDDLE_BLOCK]]:
+; VF4IC4-NEXT: br i1 true, label %[[E2:.*]], label %[[SCALAR_PH]]
+; VF4IC4: [[VECTOR_EARLY_EXIT]]:
+; VF4IC4-NEXT: br label %[[E1:.*]]
+; VF4IC4: [[SCALAR_PH]]:
+; VF4IC4-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 128, %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; VF4IC4-NEXT: br label %[[LOOP_HEADER:.*]]
+; VF4IC4: [[LOOP_HEADER]]:
+; VF4IC4-NEXT: [[IV:%.*]] = phi i64 [ [[INC:%.*]], %[[LOOP_LATCH:.*]] ], [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ]
+; VF4IC4-NEXT: [[GEP_SRC:%.*]] = getelementptr inbounds i32, ptr [[SRC]], i64 [[IV]]
+; VF4IC4-NEXT: [[L:%.*]] = load i32, ptr [[GEP_SRC]], align 4
+; VF4IC4-NEXT: [[C_1:%.*]] = icmp eq i32 [[L]], 10
+; VF4IC4-NEXT: br i1 [[C_1]], label %[[E1]], label %[[LOOP_LATCH]]
+; VF4IC4: [[LOOP_LATCH]]:
+; VF4IC4-NEXT: [[INC]] = add nuw i64 [[IV]], 1
+; VF4IC4-NEXT: [[C_2:%.*]] = icmp eq i64 [[INC]], 128
+; VF4IC4-NEXT: br i1 [[C_2]], label %[[E2]], label %[[LOOP_HEADER]], !llvm.loop [[LOOP3:![0-9]+]]
+; VF4IC4: [[E1]]:
+; VF4IC4-NEXT: [[P1:%.*]] = phi i64 [ 0, %[[LOOP_HEADER]] ], [ 0, %[[VECTOR_EARLY_EXIT]] ]
+; VF4IC4-NEXT: ret i64 [[P1]]
+; VF4IC4: [[E2]]:
+; VF4IC4-NEXT: [[P2:%.*]] = phi i64 [ 1, %[[LOOP_LATCH]] ], [ 1, %[[MIDDLE_BLOCK]] ]
+; VF4IC4-NEXT: ret i64 [[P2]]
+;
+entry:
+ %src = alloca [128 x i32]
+ call void @init_mem(ptr %src)
+ br label %loop.header
+
+loop.header:
+ %iv = phi i64 [ %inc, %loop.latch ], [ 0, %entry ]
+ %gep.src = getelementptr inbounds i32, ptr %src, i64 %iv
+ %l = load i32, ptr %gep.src
+ %c.1 = icmp eq i32 %l, 10
+ br i1 %c.1, label %e1, label %loop.latch
+
+loop.latch:
+ %inc = add nuw i64 %iv, 1
+ %c.2 = icmp eq i64 %inc, 128
+ br i1 %c.2, label %e2, label %loop.header, !llvm.loop !0
+
+e1:
+ %p1 = phi i64 [ 0, %loop.header ]
+ ret i64 %p1
+
+e2:
+ %p2 = phi i64 [ 1, %loop.latch ]
+ ret i64 %p2
+}
+
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.interleave.count", i32 4}
+;.
+; VF4IC4: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; VF4IC4: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
+; VF4IC4: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
+; VF4IC4: [[LOOP3]] = distinct !{[[LOOP3]], [[META2]], [[META1]]}
+;.
|
Co-authored-by: Paul Walker <[email protected]>
| @@ -10101,6 +10101,13 @@ bool LoopVectorizePass::processLoop(Loop *L) { | |||
| // Get user vectorization factor and interleave count. | |||
| ElementCount UserVF = Hints.getWidth(); | |||
| unsigned UserIC = Hints.getInterleave(); | |||
| if (LVL.hasUncountableEarlyExit() && UserIC != 1 && | |||
| !VectorizerParams::isInterleaveForced()) { | |||
There was a problem hiding this comment.
Based on #133099 (review) Florian would prefer not to be able to override this. Do we have tests that specifically care about interleaving? or can the restriction be removed?
There was a problem hiding this comment.
I think the problem then is that it becomes extremely difficult to write IR tests for any fixes. Any fixes would have to be unit tests.
There was a problem hiding this comment.
For now this should be fine, if interleaving isn't fixed before the release-branch we should disable for all cases where UserIC != 1.
paulwalker-arm
approved these changes
Jun 26, 2025
fhahn
approved these changes
Jun 26, 2025
| @@ -10101,6 +10101,13 @@ bool LoopVectorizePass::processLoop(Loop *L) { | |||
| // Get user vectorization factor and interleave count. | |||
| ElementCount UserVF = Hints.getWidth(); | |||
| unsigned UserIC = Hints.getInterleave(); | |||
| if (LVL.hasUncountableEarlyExit() && UserIC != 1 && | |||
| !VectorizerParams::isInterleaveForced()) { | |||
There was a problem hiding this comment.
For now this should be fine, if interleaving isn't fixed before the release-branch we should disable for all cases where UserIC != 1.
|
I suspect I've broken a post-commit build because I've forgotten (again) to add "REQUIRES: asserts". :face_palm. I'll fix it asap. |
may be better to check the remark, if at all; they should also be available w/o assertions |
david-arm
added a commit
to david-arm/llvm-project
that referenced
this pull request
Jun 27, 2025
david-arm
added a commit
that referenced
this pull request
Jun 27, 2025
Yes, that's probably even better! I hadn't thought of that. I've already committed a patch to add "REQUIRES: asserts" to fix the build quickly, but I'm happy to follow this up with a new patch to check the remark. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently if the user enables interleaving during vectorisation of uncountable early exit loops via the interleave_count pragma and the enable-early-exit-vectorization option, it will miscompile. There is ongoing work to fix this, but for now it seems safer to ignore the hint until it is supported.