Agree with @Caketray, it is very difficult to make out context from backlinks. I think the current implementation is similar to showing search results.
I feel true power of backlinks only comes with Context.
Probably some kind of compromise as @caketray mentioned ? Like checking if the current line is a header or a list item and showing the elements that nested below it ?