I can sympathize with you, but I can tell you do not worry.
Your son seems fine and perfectly normal.
At this age... they do not yet concretely make "best friends" nor keep them from here on out. Friends changes all the time, for boys or girls. And in the next year, it will be a different mix of friends and classroom.
That is how my daughter was in Kindergarten. She's now in 1st grade, and she has different friends. BUT... on MY part, I keep in touch with the Moms of her Kindergarten friends, and whom I know my daughter 'still' considers as friends, and I make play-dates for them to "keep in touch" even though they do not have each other in the same class this year. I "nurture" the "good Moms" and the "good" friends of my daughter. It is the upkeep of their social network... if not for now, then later as they get older.
My daughter (who is VERY girly) at one time, only played with a boy, her 'best' friend at the time. So what. That didn't mean she had a gender issue. It's just kids. And we still get together for play-dates and socials, even though they are in different classes/Teachers now. His mom is great and we get along too.
The thing is, nurturing friends and what not, is also what you can take part in. While respecting that a child will change all the time in "who" they play with.
My daughter tells me all the time "your'e a cool Mom.... you let my friends come over....and they are nice." She KNOWS what "type" of friends are good or bad... and she can make decisions about it. She has good social judgment... which to me, is a better ability than how many friends they have or just copying other kids..... it is quality over quantity... and social acuity versus gender based.
Your son seems to KNOW in himself, that the other boys are too rough, and he does not like it. Good for him. Yes, boys can be rough... and even my friend's son complained about that to her. He didn't like the boys in his class, because they were indeed "trouble-makers." Even I did not really like those boys either. And, I wouldn't want my daughter to be friends with them either.
My daughter is now in 1st grade, and her class is full of trouble-making meddling kids. She knows this, I know this, the Teacher knows this. My daughter will say very confidently "I don't have friends in class this year... they are not kids I want to be friends with, they are not true friends, only 'sassy'... next year will be better." And that is that. She is happy in her class and says so, she is fine with the mix of kids but "knows" these are not the kind of friends to take home to Mommy but she plays with them and does not have social problems, she knows that she can navigate herself amongst them, she knows she is fine and okay about the whole thing, and she is happy going to school, despite the mix of kids. I know, because we talk about it... and it really does not bother her that the bunch of kids in her current classroom are "yucky" according to her. I observe this as well.
So... its about the QUALITY/character of a child's friends...versus the gender of the friend or the 'quantity' of friends they have.
At at this age, it doesn't matter to them what gender a friend is. Later as a teen, they get more "clique-ish" about it and then start to separate according to boys or girls. Just like at a school dance where all the boys line up on one side of the gym and the girls hang out on the other side, while giggling and gossiping about each boy or girl.
It's all developmental. It does NOT make or damage a child.
I don't think there is a concern or problem. But, its GREAT that your son knows himself and "WHO" is yucky or nice.
That is a good sense of social analytical skill, that he has. He has "discernment" about people. That is good.
Boys do NOT have to be rough-n-tumble, or play football or soccer or, play with trucks or talk like cave-men to be a "boy." They can also be articulate emotionally, love music and art and other hobbies. What about Bill Gates? Yo-Yo Ma? Einstein? They are not typical and are not the stereotypical "macho" men. Does that make them "less" of a man? Heck no, they are so successful and follow their hearts and talents and know who they are.
All the best,
Susan