I would try not to be a mama bear ("Nothing's wrong with MY kid!").
(Why did people start calling this situation a "mama bear"? Mama bears growl at people they perceive of as threatening, but they also cuff their kiddies a lot and have no patience when their babies don't want to be sent on their grownup way!)
You can be a mama bear in this way: what other parents complain about is not the issue. Let the other bears alone, and don't listen to them. Think about what is best for your son. Don't let the issue of the teacher's style or personality cloud the air.
Whether you like the people or not, you have an opinion (from the teacher) and a second opinion (from the counselor). Where could you go for a third? You need to go to somebody you like and trust, because if that person tells you the same thing, you need to go for it.
If, as it happens, your son is having a problem focusing *in class* right now, that certainly doesn't put a "loser" label on him. It just means it's up to you to take whatever steps are necessary so that he can learn to focus and succeed. When my children needed extra help in math, they went to a math tutor, even though they didn't want to and I was sorry they had to. When a couple of them had trouble focusing in a way different from your son, they ended up going to the eye doctor and getting glasses. If a child struggles with anything too long and really falls behind, it's harder and harder for him or her as time goes by. Catch-up time should be, generally, as soon as possible.
And there isn't anything in the two recommendations against *you*, as far as I can tell. Children are like surprise packages; they're wonderful but you never quite know what's going to come out next.
I think that if I were in this situation I'd step outside the school circle and into another circle that I knew had wisdom about children. I'd call my family doctor and discuss it. In fact, I'd pick his or her brains.