Inviting the whole class is a burden for many families. Parents are tired of having a party every weekend and shopping for all those gifts, and hosts are tired of overspending for so much food, activities, goody bags, and so on. Four-year-olds are not very interested in watching another child open all kinds of toys that they don't get to play with. The more kids there are, the longer everyone has to sit there and watch. And the birthday child isn't old enough to really show appreciation (especially for gifts that they aren't as thrilled with). So it's an exercise in frustration for everyone.
We started our son with friends at parties at age 4 - he invited 4 friends. Not their siblings, not their parents. The party lasted about 90 minutes I think. Our son received 4 gifts. We played some party games and had cupcakes. The rest of the time, they ran around the yard. It was fun, manageable, and so on. We did not usually accept invitations for huge parties unless my son was particularly good friends with the birthday child. He didn't usually enjoy those big Chuck E Cheese parties anyway - a big free-for-all and a huge expense for the party-givers. So we always stayed very reasonable and he has no regrets.