Skip to content

[41.0.0] Cranelift: x64: fix user-controlled recursion in cmp emission. (#12333)#12342

Merged
cfallin merged 1 commit intobytecodealliance:release-41.0.0from
cfallin:icmp-41.0
Jan 14, 2026
Merged

[41.0.0] Cranelift: x64: fix user-controlled recursion in cmp emission. (#12333)#12342
cfallin merged 1 commit intobytecodealliance:release-41.0.0from
cfallin:icmp-41.0

Conversation

@cfallin
Copy link
Member

@cfallin cfallin commented Jan 13, 2026

  • Cranelift: x64: fix user-controlled recursion in cmp emission.

We had a set of rules introduced in #11097 that attempted to optimize the case of testing the result of an icmp for a nonzero value. This allowed optimization of, for example, (((x == 0) == 0) == 0 ...) to a single level, either x == 0 or x != 0 depending on even/odd nesting depth.

Unfortunately this kind of recursion in the backend has a depth bounded only by the user input, hence creates a DoS vulnerability: the wrong kind of compiler input can cause a stack overflow in Cranelift at compilation time. This case is reachable from Wasmtime's Wasm frontend via the i32.eqz operator (for example) as well.

Ideally, this kind of deep rewrite is best done in our mid-end optimizer, where we think carefully about bounds for recursive rewrites. The left-hand sides for the backend rules should really be fixed shapes that correspond to machine instructions, rather than ad-hoc peephole optimizations in their own right.

This fix thus simply removes the recursion case that causes the blowup. The patch includes two tests: one with optimizations disabled, showing correct compilation (without the fix, this case fails to compile with a stack overflow), and one with optimizations enabled, showing that the mid-end properly cleans up the nested expression and we get the expected one-level result anyway.

  • Preserve codegen on branches.

This change works by splitting a rule so that the entry point used by brif lowering can still peel off one layer of icmp and emit it directly, without entering the unbounded structural recursion.

It also adds a mid-end rule to catch one case that we were previously catching in the backend only: fcmp(...) != 0.

…odealliance#12333)

* Cranelift: x64: fix user-controlled recursion in cmp emission.

We had a set of rules introduced in bytecodealliance#11097 that attempted to optimize
the case of testing the result of an `icmp` for a nonzero value. This
allowed optimization of, for example, `(((x == 0) == 0) == 0 ...)` to
a single level, either `x == 0` or `x != 0` depending on even/odd
nesting depth.

Unfortunately this kind of recursion in the backend has a depth
bounded only by the user input, hence creates a DoS vulnerability: the
wrong kind of compiler input can cause a stack overflow in Cranelift
at compilation time. This case is reachable from Wasmtime's Wasm
frontend via the `i32.eqz` operator (for example) as well.

Ideally, this kind of deep rewrite is best done in our mid-end
optimizer, where we think carefully about bounds for recursive
rewrites. The left-hand sides for the backend rules should really be
fixed shapes that correspond to machine instructions, rather than
ad-hoc peephole optimizations in their own right.

This fix thus simply removes the recursion case that causes the
blowup. The patch includes two tests: one with optimizations disabled,
showing correct compilation (without the fix, this case fails to
compile with a stack overflow), and one with optimizations enabled,
showing that the mid-end properly cleans up the nested expression and
we get the expected one-level result anyway.

* Preserve codegen on branches.

This change works by splitting a rule so that the entry point used by
`brif` lowering can still peel off one layer of `icmp` and emit it
directly, without entering the unbounded structural recursion.

It also adds a mid-end rule to catch one case that we were previously
catching in the backend only: `fcmp(...) != 0`.
@cfallin cfallin requested a review from alexcrichton January 13, 2026 23:47
@cfallin cfallin requested a review from a team as a code owner January 13, 2026 23:47
@cfallin cfallin enabled auto-merge (squash) January 13, 2026 23:48
@cfallin cfallin merged commit 4a67a85 into bytecodealliance:release-41.0.0 Jan 14, 2026
174 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants