I've never swum in my contacts. I just got new contacts last week, and noticed that the instructions say to remove them before swimming. My doctor says that the issue isn't just losing them; it's the fact that the chlorine and chemicals in pool water (or salt water, in the ocean) will definitely affect and damage the lenses themselves even in one session of swimming, and even if you keep your eyes shut. If you want your lenses to last, don't swim in them.
I would also be concerned about the lenses, which are very moist and adhering directly to my eyeball, holding the chemicals from the pool in my eye the rest of the day after swimming. I know the pool water doesn't stay "between" the lens and the eye, but today's lenses are very moist and permeable for comfort, and I would think that would mean they would retain moisture from being in the pool -- even with eyes closed, some water is still getting into the eyes.
My husband swims all the time and wears his glasses right up until he gets into the pool, then puts them inside his shoes so they won't get stepped on. He says once he's in the pool he seems to see fine for swimming. Same with me -once I'm in the water I don't really notice my bad vision as much.