Our daughter was born May 3rd, and we took her to two different memorial day parties. For both, we dressed her lightly, kept a blanket nearby, kept her hydrated and kept the shades down on her stroller. We had some access to the hosts' homes in both cases. While they were primarily outside parties, no one had an issue with us bringing our daughter inside when it got too hot or she fell asleep. She also attended graduation parties, July 4 bbq's, labor day festivities, and several other parties over the summer - all without incident.
If you are prepared, there is no reason to stay home. I am a huge advocate of taking the baby out, early. Living in a bubble does not help you, and it does not benefit your child as much as some would have you belive. I have a very healthy, very social, and very happy 14 month old now. She is a generally good natured kid, but I believe part of it is that we have never excluded her from social events. She has gone to parties and restaurants since she was a few weeks old, and she continues to go. She is learning good public behavior at an early age. It is never too early to expose kids to the rest of _our_ lives -- after all, it is going to be a part of their life too.
I'll get off my soapbox - it won't hurt your baby to go as long as you are prepared. Make the outing short if you are nervous.