Can’t say I’ve had that experience but I think my windshield deflects a lot of the crap that gets thrown up. It gets much dirtier than my visor ever does. And while it is a rare (and it might depend on your geographic location) it is possible to get ‘dirty’ rain. If there is dust/dirt in the atmosphere it can get washed to the ground by rain. Usually easy to tell after the fact as you can see the dirt on cars once the water has evaporated.
But light rain by itself can be a PITA. Very soon after I started riding I was out in a very fine light rain in an urban area, lots of shops, parked cars, cars coming out of side streets hidden by the parked cars. That light rain just coated my visor and I couldn’t see anything. If I had been going 60mph it would have blown off but at 20mph, it just stuck to the visor. And with all the traffic and hidden dangers I really needed good visibility. I ended up just riding with my visor up. Not too bad at 20mph but it was a rude introduction into the world of riding in poor visibility.