Not at all! We got to the pool every week during the winter and my kids LOVE it! Plus, it's a great way to break up the normal winter activities with something warm. My daughter's 4th birthday party was at an indoor pool and I did have every parent stay but I also had lined up some other family members of ours to don suits and get in to help out. I would think if it's a preschool aged party, most parents would probably stay with their kids at a pool, but I would indicate on the invitation whether or not you want them to stay, but as I said, have a back up plan if someone can't stay or refuses to put on a suit (which some moms might do, sadly).
The only bad part about this is that you should have posted it a week ago! I just bought a bunch of super awesome pool supplies that were on super clearance at Target and Kmart. Kmart might still have some stuff, but Target was pretty cleared out, although possibly worth a look. I would look on line at those stores maybe and see if you can pre order some super cool party favors for cheap! I got goggles at Kmart for like $1-2 each and they were nice goggles. I also got some balls and dive rings. For my daughter's party I also ordered the mini beach balls from Oriental Trading Co. They come a dozen in a package and they are only like 30 cents a piece or something. You can also order buckets and stuff from them too. They also have summer clearance too but not always on the pails and such.