I can make it happen pretty reliably now, although not from scratch.
Have pasted images in notes that are in a subfolder - images are in that subfolder as well
Create a note in the root folder
Paste an image in this new note, then CTRL+Z because you changed your mind (wrong image for example).
The link is now gone from the file, but the new file you pasted still exists in the file browser.
If I look at the file in the internal Obsidian browser, its the equivalent image from a previous note (let’s say Pasted Image 2).
If i go and have a look at it in Windows explorer, its the image I pasted last.
Some kind of weird caching of images going on? Some issue with CTRL+Z? I would expect CTRL+z to completely remove the image I just pasted, not just the link.
Will try to reproduce from scratch in a new vault.