You cannot programmatically focus an element after removing its focus handler #4867

Description

I'm seeing this in the latest chromium browsers Windows Chrome & Edge 89, using jquery 3.6.

If you add and remove an element's focus handler, subsequent calls to $element.focus(), will do nothing.
HTML:

<div tabindex="-1" class="fdiv"></div>

CSS:

.fdiv{
  height:100px;
  width:100px;
  border:1px solid black;
}
.fdiv:focus{
  background-color:green;
}

See the linked to test case below. I have a simple div that turns green when you focus it and some js that will focus said div after 2 seconds.
Before setting this timeout I add and then immediately remove an empty focus handler:

//Using https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.0/jquery.min.js

const $origDiv = $('.fdiv');

// Comment out the following line to fix
$origDiv.on('focus',function(){}).off('focus');

setTimeout(function(){
	$origDiv.focus();
},2000)

If you run the fiddle, the div should turn green (gain focus) after 2 seconds, but it doesn't!
If you comment out the focus handler line, all works as expected.

Link to test case

https://jsfiddle.net/JoolsCaesar/ut371acd/

^{EDIT by @mgol: I fixed syntax highlighting in code blocks.}