I have had 3 children so far. During each, I turned down the blood screen. My reasoning was that if there was an abnormality, the types that they usually check for can also be discovered during an ultrasound. (The tech checks the brain, heart, liver, stomach, etc, etc, etc.) If the ultrasound discovered something, then my husband and I would pursue other options.
With my second, the ultrasound tech discovered that my son had "bubbles" in his brain. It was an indicator for Down's Syndrome, but it also was found in completely healthy children. We had another ultrasound done a few weeks later, and the "bubbles" were gone. I asked my doctor about if it would have appeared on the blood test, and he said maybe, but that an ultrasound is a much better predictor of any potential abnormalities. (Plus you can see your baby's heartbeat, movements, and cute features already!)
I was a high risk pregnancy for all three children because of a genetic kidney disease. The blood test becomes "standard" during high risk pregnancies. If all else is going well, I say don't take the test. Make sure things check out during the ultrasound, but beyond that, relax and have a great pregnancy!