We need to see some examples of your task, both to be able to reproduce your issue, and to help you understand what is going on.
But if you’re grouping by file.link then that could hinder sorting outside of that group. And when you’re sorting on the modification time, it doesn’t sort on anything else…
Finally, if the priority is given for the entire file, that might also affect stuff.
So please provide some more information, to enable some of us to possibly help you any further.
Bonus tip: How to present code properly in a forum post
If you want to showcase either markdown, or code blocks, or dataview queries properly in a forum post, be sure to add one line before and one life after what you want to present with four backticks, ````. This will ensure that any other backticks (like for code blocks) is properly shown.