Graph View shows all the tags even though I only filter one Tag

Have a look here Exclude a specific tag from the graph view - #13 by Naraulki Haven’t tried this (yet) myself, but I think could be adapted into a solution to your problem.