Bug: Middle mouse button: Click for popup, click and hold for Pan

If I quickly click the middle mouse button no pop-up pops up, I have to click and hold for a very short time for it to pop up. Panning works as expected. But it seems like a glitch that the pop-up is activated on fast clicks.
(V5 has the same behaviour)