[41.0.0] Cranelift: x64: fix user-controlled recursion in cmp emission. (#12333)#12342
Merged
cfallin merged 1 commit intobytecodealliance:release-41.0.0from Jan 14, 2026
Merged
Conversation
…odealliance#12333) * Cranelift: x64: fix user-controlled recursion in cmp emission. We had a set of rules introduced in bytecodealliance#11097 that attempted to optimize the case of testing the result of an `icmp` for a nonzero value. This allowed optimization of, for example, `(((x == 0) == 0) == 0 ...)` to a single level, either `x == 0` or `x != 0` depending on even/odd nesting depth. Unfortunately this kind of recursion in the backend has a depth bounded only by the user input, hence creates a DoS vulnerability: the wrong kind of compiler input can cause a stack overflow in Cranelift at compilation time. This case is reachable from Wasmtime's Wasm frontend via the `i32.eqz` operator (for example) as well. Ideally, this kind of deep rewrite is best done in our mid-end optimizer, where we think carefully about bounds for recursive rewrites. The left-hand sides for the backend rules should really be fixed shapes that correspond to machine instructions, rather than ad-hoc peephole optimizations in their own right. This fix thus simply removes the recursion case that causes the blowup. The patch includes two tests: one with optimizations disabled, showing correct compilation (without the fix, this case fails to compile with a stack overflow), and one with optimizations enabled, showing that the mid-end properly cleans up the nested expression and we get the expected one-level result anyway. * Preserve codegen on branches. This change works by splitting a rule so that the entry point used by `brif` lowering can still peel off one layer of `icmp` and emit it directly, without entering the unbounded structural recursion. It also adds a mid-end rule to catch one case that we were previously catching in the backend only: `fcmp(...) != 0`.
alexcrichton
approved these changes
Jan 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We had a set of rules introduced in #11097 that attempted to optimize the case of testing the result of an
icmpfor a nonzero value. This allowed optimization of, for example,(((x == 0) == 0) == 0 ...)to a single level, eitherx == 0orx != 0depending on even/odd nesting depth.Unfortunately this kind of recursion in the backend has a depth bounded only by the user input, hence creates a DoS vulnerability: the wrong kind of compiler input can cause a stack overflow in Cranelift at compilation time. This case is reachable from Wasmtime's Wasm frontend via the
i32.eqzoperator (for example) as well.Ideally, this kind of deep rewrite is best done in our mid-end optimizer, where we think carefully about bounds for recursive rewrites. The left-hand sides for the backend rules should really be fixed shapes that correspond to machine instructions, rather than ad-hoc peephole optimizations in their own right.
This fix thus simply removes the recursion case that causes the blowup. The patch includes two tests: one with optimizations disabled, showing correct compilation (without the fix, this case fails to compile with a stack overflow), and one with optimizations enabled, showing that the mid-end properly cleans up the nested expression and we get the expected one-level result anyway.
This change works by splitting a rule so that the entry point used by
briflowering can still peel off one layer oficmpand emit it directly, without entering the unbounded structural recursion.It also adds a mid-end rule to catch one case that we were previously catching in the backend only:
fcmp(...) != 0.