If your reasons are truly out of concern for your baby, no, I do not think it is wrong for you not to want your baby over at your in-laws house. But if your reasons are more so because you are upset with your in-laws, either because of their behavior towards you or because they haven't "made the effort" to come and see the baby at your parents' house, then yes, I think you could make an effort to let them see the baby on their own turf.
I obviously don't know your family, but I would venture a guess that if you lived in your own house, this would not be an issue. I think it's probably more related to you living with your parents and your in-law's feeling like that's not really "neutral" territory. They might be thinking "Why do your parents get to spend all the time with the baby and you won't even make the effort to take the baby to see them?" I mean, let's face it, your parents don't have to go further than a few feet out their bedroom door to see their grandbaby every single day. Is it fair to expect the other grandparents not to be a little jealous of that arrangement?
I mean, maybe they don't really care, but it's just hard to say one way or the other. I think it would be a nice gesture to take the baby over to their house, even if just for one visit. They probably won't ever feel comfortable going to your parents' house if they don't already, but you can't really hold that against them. Families are funny that way. My mom went to my mother-in-law's house on several occasions over the years before my MIL passed away, but I don't think my mother-in-law went to my mother's house even one time in the 16 years they knew each other, even though she was invited on many occasions.
I hope you are able to find a way to work around your reservations regarding your in-laws. Babies need lots of love and most grandmas and grandpas have lots of love to give.
Blessings,
N.