First of - relax. I asked my doctor about this when I was pregnant. I am 37 and I have lived with indoor/outdoor cats (only one to three at a time) all of my life and changed catboxes since I was 10. I am not sure if I have been exposed to it or not (probably have) since my doctor just said "if you've had cats that long and you garden as well, you probably have come into contact and no need to do a test". I didn't know about toxoplasmosis until I was 7 months pregnant, so it was kind of an "oh well" issue with him (not a nice person). My babies are both happy and even a bit advanced (but I'm a biased Mom).
Yes, toxoplasmosis is real. You can have one or both of two tests done. You can have your cat tested. If the cat has it, get tested. If the cat does not and you have had cats before this one, still get yourself tested. You can have your doctor test you for antibodies. If you have the antibodies, it means one or more of your cats has it. It also means that you CAN change the litter. Coming into contact with it for the FIRST time WHILE you are pregnant is the problem.
If your test shows that you do not have antibodies to it, DO NOT deal with the litter AT ALL: no scooping, no changing, nothing. If your cat has access to the outside or outdoor cats, he can still catch it and bring it home. Not too sure about how it is passed, but don't let your cat lick your face, since if it's in the litter maybe it can pass from kitty kisses?
Just ask your doctor before freaking out about it. Here is a website to have your Hubby read this link on Wikipedia for you. I was always reading things and gave myself a few unneccessary scares, so I started to have my husband screen some things. http://en.wikipedia.org/wiki/Toxoplasmosis
This has some really mind-comforting information. But if you're the type to worry about things a lot, have someone who isn't read it for you.