M.M.
Put your foot down! Just because everyone wants to hold the baby doesn't mean that it's a good time.
I know how hard it can be to put your foot down to family when you've just become a new mommy and may not be sure of yourself (at least that was MY experience with my inlaws) but now is a good time to start!
If your baby is on a regular schedule, do your absolute best to keep him on that schedule. My advice with the baby in general, is to not worry too much, really. Babies can handle what they can handle, and can't handle what they can't handle. I think the best plan, even for the long run, is to expose baby to lots of new experiences "out of the norm."
That's just my own personal view though. We took our son camping when he was a mere 2 months. He was an angel right up until the ride home when I needed to pull over and let him scream for an hour. Hey, it's not easy being a new person! However, my son now, at eight months, is pretty adjustable.
Just say no when you need to. Your son will let you know what he needs. Think positive thoughts!!! And good luck!
***edited to add***
When we went camping, I insisted we bring the swing with us. At home it was the only surefire way to get him to sleep and I wasn't about to end up in the woods with a screaming baby! The swing was the first thing we set up and he conked out in it, right in the middle of an open field like nothing...
What I mean is, if your son has a particular attachment to something (the swing, a blanket, being driven around to sleep) make sure to bring that stuff with you!