I wouldn't go if I were you. It might make things worse with his family if they see that you're uncomfortable being there.
As far as his family not liking you, I can relate. I don't know how open they are about it, but my husband's family is kind of passive-agressive about it. Particularly his mother; I hear mean things she has said to others about me, but she seems to act nice around me. She never makes eye contact with me and brushes off everything I say as if it's nonsense.
I finally said something about it to my husband and he gave me the best advice ever. He said to pretend I have no idea that she doesn't like me and that I never notice how she treats me. Since then, it still bothers me, but I don't feel awkward around them any more, I feel like I'm on to her and she can't do anything to ruffle my feathers.
I don't know if your situation is similar at all, but if it is, I hope that helps.