If the number of times the kids use the pool is correct, I'd say you'd be dealing with this guy whether you had a pool or not. I hate to say it, but I think some older people sometimes have just too much time on their hands and too many years have past since they had young people around. I think it's a case of the "old crumudgeon" and he's picked your house to be his pet project.
I'd ask this guy to produce a hard copy of this "poll". I'll bet you anything, no one has been polled, and it's probably one other neighbor that commented in passing to this guy about noise. I think he's grasping at straws, and if he can get your goat in the process all the better.
To be on the safe side, find out what your city's noise ordinances are, and be sure you're not violating them in any way. Also, if you live in a town-house community, get a copy of your neighborhood by-laws and make sure you're within what's acceptible noise-wise by your association. If you're all doing what the city deems as acceptible, no poll or any other neighbor complaint will "hold water".
I have a neighbor who did this to us about our dogs. He claimed they barked day and night and would call the cops on us all the time. Though our dogs are 2 big hunting dogs, they are in the house all day except for when we take them for walks.
The cops started getting wise to his antics. Unbeknownst to us, they'd sit outside our house and monitor for 5 to 15 minute intervals to see if the dogs were really nuisance barking by the city's code/definition. (Something like continuous barking for 5 to 10 minutes over an hours time or whatever...you get the idea). We had no idea he had been calling the cops until one came to the door to let us know and give us a warning.
The officer was quite annoyed with the neighbor, not us. And he made a point of saying the warning was protocol because they had been called to the house at least 5 times that week. He said the dogs weren't barking, but he had to give us the warning because of the number of times they were called by this neighbor to our address.
After the warning, my husband went to all of the neighbors to see if our dogs were bothering them too. We were shocked to find out the guy was doing this to all of us! All of us had indoor dogs, except for one guy down the street who left his Weimereiners outside all day. I think they were the ones making all the noise. The cops finally stopped coming for all of us and decided the guy was a lonley old Kook kicking up dirt for some entertainment! This guy did other weird stuff too. But that's another story.
I guess I'd say find all the facts out for yourself first. Then get bummed later...or not at all!:) I'll bet it's not as he's making it seem.
Good luck.