To style unresolved links you’ll need a CSS snippet that targets the .is-unresolved class. It used to only work in Reading Mode; I don’t know if that’s still true.
The CSS Cawlin linked to above works fine in Reading mode, but still no change for Source/Live Preview modes.
There are variables now that cover both Editing and Reading modes that make things a bit easier depending on the theme you are using. These are from Obsidian’s default CSS.