I have to agree with Jen C. about urine being sterile. I was surprised when my son was born a few years ago and I learned that many people don't use wipes on babies if they only have a wet (not messy) diaper! The alcohol and other stuff in the wipes is harsher on their skin than the urine. No powder, either, because it's bad for their lungs. I tried it and we've had no ill effects, hardly any diaper rash ever (son is almost 4 now and potty-trained for a while).
What bothers me more is the thought of guys not even wiping - I had that debate with my husband when we potty-trained our son. He REFUSES to let our son be a "dabber". I insist that at least he wipe the rim of the toilet if he drips. And my hubby has been good about teaching the little guy to put the seat and lid back down, QUIETLY.
Back to hand washing, some of the soap in commercial restrooms is terrible on your hands. I prefer moisturizing hand sanitizer and have some with me all the time. Although I don't really like the proliferation of antibacterial products everywhere. I think it is contributing to bacterial resistance to antibiotics. Also, it helps you keep a strong immune system to have some exposure to everyday germs.
That said, there is one strain of hepatitis that is transmitted by having fecal material on your hands (such as not washing after the bathroom). Of course, some other nasty things as well (E. coli for instance). And if you don't know if other people have washed their hands, those kinds of germs could be on the door handle even if you didn't get anything on yourself when you went. So another rationale for hand washing is that bathrooms can just be germ-y places, regardless of how clean YOU are.
So even though I lean more toward Jen C's perspective, I have to agree that you are probably in the majority with your opinion. Unfortunately, public restrooms gross people out in general and that could be part of why they hesitate to wash in dirty sinks... there is just no way to win this one.