We are in the same boat as you when it comes to too many kid groups - church, scouts, sports, school old and new, neighborhood, my friends with kids, and not to mention family and all the cousins, but tbh, I just send out an email blast every year and whoever shows up shows up. I'm one of those who would rather not offend anyone and invite everyone, than stress over the time it will take and speculating who will be offended by deciding who to invite.
The big parties are just backyard parties with picnic games, a fully loaded pinata (to fill their favor bags) and a cake. Its never the organized event like most kid parties, although I do try to get creative every once in a while (one year my son liked frogs so I made the kids do an obstacle course hopping like a frog and then we had a contest of who had the best "ribbit".)
The smaller parties are just for family but then we have a few cousins of the same age where if we go to a place like Main Event my son can still have fun without having to deal with too many grown ups.
Hope it all turns out!