Just putting this here for others as a PSA. I was fortunate to find an alternative Quick Switcher plugin called, “Another Quick Switcher” in Community Plugins that solved all my problems. It lets you use Quick Switcher sorted by File modified date and not limited to only 10 results.
Shout-out to tadashi-aikawa for writing this plugin!