So sorry this is happening to you. Unfortunately, this will not be the only time in your life that you will be humbled! Children go through good and ugly phases, in their lives. I hope this season passes quickly for you. But it needs your complete and united support from both you and dad, to nip this swiftly! Don't worry about justifying it with the other parents. They just want to know that it is being addressed. YOU are not a bad mother!!!!
He has learned this behavior somewhere, he has seen it somewhere. Is he playing with older children? Watching age-appropriate TV? How is his speech? I personally would make sure that you talk to him about "others touching him" and what's not appropriate. Make sure you are letting him know that, should ANYONE ever touch him in his private area (anything covered by a bathing suit - boy or girl) he needs to tell you. Also tell him that ANY TOUCH that makes him uncomfortable, he can tell that person, firmly "DON'T TOUCH!"
He needs specific guidance. He needs to know exactly what is acceptable and what is not. This is not just a boy-girl thing. Touching anyone, in any manner in school is inappropriate. Some of this behavior is age related. (possible hugging and kissing). Spitting and touching someone's private area (i.e. butt) is never unacceptable. Sitting him down when the situation is not enflamed, in private, is best. Having a united front with Dad is necessary. Letting him know that there will be consequences to his action (you do not need to spell out these consequences - just yet.) If he's pushing the boundaries and getting away with it, strengthen the boundaries and find a better way to end his negative behavior. He may be getting away with it because he can. Be consistent. Yes, it can get tiring! He may be seeking attention and negative attention is still filling his need for attention (i.e. bothering his brother, inappropriate touching, etc...) You need to find his "achilles heel" (i.e. specific appropriate punishment for your son - the one that makes him squirm.) It sounds like time-outs don't phase him. For example, my children hated writing assignments - "I'm sorry" letters, they hated apologies to parents face-to-face, etc... Make this child accountable. It's not a matter of him "showing respect" for girls -it's what is acceptable behavior for all people. Period. Positive reinforcement when his behavior begins to change is important. He may at some point recognize that he likes positive attention better.
If you continue to see this behavioral problem, please seek a professional. A counselor or your pediatrician will work with you to address these concerns.
Good luck.