Scroll callout box only on KEY+mouse wheel

What I’m trying to do

I was able to implement a scrollable callout box as detailed in this thread:

How to make callouts a fixed height and then you can scroll the contents inside the callout?

Now, I have a Daily Notes template with several callout boxes with a task query in each, and while I want to keep the scrolling behavior, it’s also annoying when scrolling to traverse the (long) page and activating the callout scroll box. I use a split screen view that leaves little room on either side to place the cursor while scrolling the page to avoid activating the scroll inside the box.

Is there any way to require using CTRL some keypress before the callout box will be allowed to scroll? This would solve my issue in an elegant way.

Things I have tried

I did not find this idea anywhere in the forums or help docs. I understand the keydown event and listener requires Javascript, is there any way to implement this into the custom css file I have for scrollable callout box?