M.D.
I know how you feel.
For my daughter's last b-day, she wanted a Teddy Bear-making party and it was going to cost $20-$25 a guest, so it was important for us to have a small group of guests. I told my daughter that she can invite 6 friends for her 6th birthday party.
My daughter has a very good friend who's 4 year-old brother tags along a lot.
So, I told this girl's mom that "We've decided to invite only 6 kids this year." I also told her that all the guests are in 1st grade. This was clear enough for this mom to understand that her 4 year-old son was not invited.
At the party, I overheard this mom talking with another mom saying "My little guy really wanted to come too." But luckly, I heared the other mom say "Yeah, my little one wanted to come too. But, I think it is good that the kids have friends of their own and separate lives from each other. My little one will get invited to a party by his friends one day where the big guy can't go." Then I heared the original mom say "You're right."
Good luck to you and hopefully there will be no hard feelings!