I totally and completely understand what you are experiencing, as my husband does this a bit as well, although maybe not quite as much as you describe... (eta: my husband does not do this in front of anyone, or in public...)
And I mostly agree with Theresa N. on this one. I do think that men have different ways of showing love/affection/attraction than we women do, and different expressions mean different things to us. I KNOW that my husband equates my desire for him, my interest in him sexually, and my physical expressions of attraction to my LOVE for him, if that makes sense. If I'm more sexually active with him, he feels more loved. If I'm more touchy feely with him, he feels more loved. It doesn't matter nearly as much to him if I tell him how much he means to me, or if I do things for him to show him how much I appreciate him, as it does when I am physically/sexually close to him. Men are (generally) wired differently than women in that way.
For me, I don't interpret his walking up behind me while I'm putting dishes in the dishwasher, kissing me on the neck, then proceeding to put his hand down the front of my pants as "love." I usually loathe it. It does not have his desired effect, whatsoever. But while I personally don't like what he's doing, I DO understand his intention. That IS his way of expressing his love for and attraction to me.
I don't really have a good response when he does this, although I usually try not to be TOO admonishing... I will usually say a slightly playful "knock it off!" or something like that...
I have talked to him about this, and he does hear me, and doesn't get defensive... but I honestly don't think he understands or realizes how I feel when he does that stuff, because the way I feel IS NOT the way he'd feel if I were doing it to him... that's why I don't do what was suggested, to do the same thing to him at a time that's perhaps less convenient for him. My husband would LOVE it no matter when I did it to him. So that tactic would not work at all for me. I think the better way to handle it is to be honest about how it makes you feel and WHY.
I have said to my husband in the moment, when say, he hugs me, then puts his hands up my shirt, "Can we ever just hug or kiss, and you not grab my boobs or put your hands down my pants?" Followed by, "It makes me shy away from hugging or kissing you when I think that EVERY time it is going to be made into something sexual. Sometimes I just want to hug you and be hugged by you, or kiss you and be kissed by you. THOSE things remind my how much you LOVE me, but don't make me feel like you ONLY want sex from me." I have also explained to him, when not in the moment, how men and women (generally) express and feel love differently. Men need sex to feel loved. Women need loved to feel sexy. (Of course that's a huge generalization, but again, as a very general rule, I think it holds basically true.) And I've gone into detail with him about that, like how when he is loving toward me, hugging me, touching me non-sexually, etc., that I feel more like being sexual with him because I'm feeling loved... and I explain to him that I understand that he needs me to be more physically/sexually expressive toward him, and that makes him feel loved... saying it that way helps him to see that neither of us is at fault for how we feel about expressions of love, and to understand better where the other is coming from. Having that understanding keeps it kind of neutral, like neither is wrong, but we're just different. So to answer your question, neither of you is 'right' or 'wrong.' You're different, and you both need to better understand each other's feelings and intentions, and communicate those things to each other...
My husband seems to understand that language, and for a while, he'll be more affectionate without all the groping. It still happens, and sometimes I'll get to the point that I have to remind him that sometimes I need to feel LOVED without feeling like his sex object, and he'll step it down for a while.
Hope that helps.