My first instinct is to say that you're being over-protective, but when I think about it my kids really haven't been too many places without us or one of our parents either.
If it were my decision, as hard as it is, I'd say let him go. Depending on where the field trip is, chances are, he'll have a great time and will have all kinds of great experiences to share with you. If it's a trip to the Museum of Science and Industry, that's a big deal, and I could certainly see you wanting to be there. If it's to a water park, and you know he's not a good swimmer, I'd say no.
But, if it's to a library or somewhere local and safe, I'd say let him go. It's hard not to be over protective of our kids and to do everything we can to keep them safe. Imagine the look on his face when he comes home to tell you about the great time he had.