I think you just need to accept that nobody is perfect. Not your kids, not my kids, none of us adults. We are all imperfect.
My younger daughter had very unclear speech at ages 2-3 (we were the only ones who could understand her, and not always). Additionally, she is tiny for her age (in the 10%, while her big sister and both parents are well above the 95%). So she has always been the recipient of comments.
I guess my take on it, as a woman who is 6' tall (I've been this tall since junior high, by the way), is that people comment on anything out of the ordinary. It's best not to pretend that nothing is out of the ordinary. Being different is okay. When people say, "Wow, you're tall! Do you play basketball?" I reply, "No. Do you play miniature golf?" and then smile sweetly. Basically I answer one dumb question with another. :)
Back to my daughter, when people would ask if her language skills were behind, I'd say, "Maybe. We're working on it. She's trying really hard to pronounce some of the sounds that are hard for her." I mean, it's true - no, nobody could understand her, and yes, she was working hard to make herself understood. Whatever, now she's 6 and has no trouble making herself understood - maybe the commentary when she was 2 led her to try harder. I don't know.
In terms of her size, she knows she is tiny. She will probably always be tiny (in a family of giants, no less). When she was about 4, she asked me if it was weird that she is so little. I told her, "No, it's not weird. This is the size you are supposed to be. It's a good thing, too, because you are SUPER cute and tiny!" So we have always acknowledged her differences, but in a positive way. What makes her different also makes her special. When people comment on her size ("How old are you? Four?") she replies, "I'm six. I'm small and very cute for my age!" It's so funny.
Anyway, I would just try not to be offended. People comment on anything unusual. It is what it is. The sooner you and your child can come to grips with it, the better off you'll both be.