Hi B.,
Last year, we enrolled my daughter (then 4) in swim classes offered by the City of Irvine. It was $50 for 4 weeks (Mon - Thurs). The first week was pure hell! My daughter was excited at first and then got really scared because she didn't know anyone. It got worse when the group she was in was all boys with the instructor being a guy. The first class, she didn't even step in the pool. Then next day she went because there was a female aid helping in that class and would only allow the female aid to help her. It was a nightmare the first week because they couldn't guarantee that a female aid would be available. The second week was better (they were able to place a female aid in the class) and she had a lot of fun. But because of that first week experience, my daughter doesn't want to take anymore swim classes. We're not going to force her because what good can come from it. However, just from that one class, she has improved in her swimming abilities. She can swim the length of the pool with no problem. However, I'm sure she'd benefit from a couple more classes and if the opportunity presents itself again, I'm sure we'll enroll her again. But, for now, we help her with her swimming until she feels comfy about taking classes again. If and when that will happen. =)
From what you wrote, maybe your son isn't ready for swimming lessons, yet, and maybe you should try again when he's older. Or, it may help if he has a friend who's taking swim classes, maybe they can enroll together. It may make him feel comfortable knowing his friend is there, too.
Good luck and hope everything works out!