Hi M.! I, personally, don't think there is an age that is too early for swim lessons. I wish I'd started my oldest (almost 11) when she was younger as she is behind her peers in knowing how to swim, and water safety was a huge concern for me for a long time. I learned a lot from that, and started my 3 1/2 year old in city swim lessons when she was 9 mths old (in the parent/toddler class). The city swim lessons here are very good, and very inexpensive. I'd definitely look into swim lessons for your kiddo this summer!
As for other activites, I don't think you can scare them off at this age, as it is just fun for them. As long as you don't become the high pressure mama, they have fun! I haven't started my 3 yo yet, but that's because her older sister already has me running! LOL I am considering dance or gymnastics for her next year though. She really wants to play soccer like her sister, but she has to wait a couple more years.