I agree with most of this. However, instead of using a hotkey to move focus over to the file explorer, why not just use the same hotkeys that move focus from pane to pane?
Assuming you have your file explorer on the far left, CMD+Left would pop focus over to it. Up and down arrows would then allow you to navigate the files.
This would be a huge improvement in my opinion.