I think if you didn't know the right answer, you wouldn't be asking this question. You already know the right thing to do.
However, if you want to take the selfish route (I am not judging, I am right there with you- you described my husband's family completely and my feelings about them! I didn't want them there either!), it is totally your decision. Having a baby is a special time for your family.
At the end of the day, though, you can't take that decision back.
Why not just limit the time? Have your husband tell them they can visit for a few hours but that's all, they have to go because your family needs time to bond with its newest member.
Ultimately, I let my in-laws visit. Yes they were annoying. But, my baby is part of their family too and I'm glad I made that decision. Also, it was out of respect for my husband.