Hardly anyone RSVP's anymore, they really don't. And, I've found even if someone says they're coming they sometimes don't. Personally I stopped even hoping people would RSVP years ago.
I've helped my sister plan my nephew's parties for years, he'll be 12 this July. The main problem with his is that it's in the summer, so it's hard to get hold of friends from school. She thought she'd circumvent the problem by holding it in June last year, before school let out, and it was still a bust. No one really RSVP'd, one friend said he thought he was coming, didn't show, then told my nephew he forgot. His best friend came and there were the usual cousins and neighbors, so he had fun. His school has the policy "give one, give to all" for invitations, if you hand them out in class. (But can't give a contact list due to privacy concerns.) So parents who don't want to invite the whole class invite in the parking lot, literally hand them out there. My sister has pretty much decided big birthday parties are over, which came as a shock to my nephew, but she's planning something just he and his cousins can do.
My guy is turning 4 soon, his party is in 2 weeks. His preschool won't allow invitations to be handed out at school, you had it good when your son was younger! So no contact lists, to invite his class friends I'll have to wait at drop off or pick up this week. He says his "whole class is coming," I know they won't. There are 2 I know will, the rest I haven't a clue. But he'll have cousins, big brother and sisters, our neighbor and 4 other friends that will be there, so he'll have fun. I've been preparing him for everyone not coming, telling him sometimes people have other things to do, he accepts it, and won't be devastated.
Being the day before Easter it would make sense that people were busy. Our party last year was the day before Easter, no friends came, just related kids. It was a bad time for several, there were a few family members who had to decline. I made it lots of fun for the kids who were there, my guy had a blast. And remember, 18 kids wouldn't have responded to your invitation or reminders, their parents are the ones who dropped the ball, IF they saw them. A manners note wouldn't change a thing, sigh. People don't think about how their actions, or lack thereof, affect others.
It sounds like you planned a great party, your son hopefully enjoyed it overall. I'd just explain to him if he still asks, that people were more then likely busy, and remind him they never said they were coming in the first place. Tuck this experience away for next year, I think a dozen or so is really the most you can realistically expect to attend, RSVP's or not, everyone can't or parents choose them not to attend 18 or so parties a year.