Hello Michelle,
I am a mother of 4 and I have dealt with a few situations just like this with my own children.
My oldest daughter, who is now 7, did not like men. It was really odd. No matter what man held her, she wasn't happy. And, obviously, as a 2 month old she couldn't tell the difference between a man and woman. I spoke with her peditrician and he said it could easily be the smell of most men. The smell of cologne, etc. Childrens senses are very strong and you might be suprised what they pick up.
My youngest son now was not happy with anyone but his dad, myself and one of his aunts. Not even my mother could make him happy and I know how stressful that can be. It even made me feel sad for the other people wanting to love on him and he just didn't want anything to do with them. We really just had to wait it out. My mom continued trying but we would never force him to her. We would have her spend smaller amounts of time with him to start. And, we would do that while we were around.. If she is one of their #1 babysitters then they might look at her and know that when they go to her, you are leaving. My son is 13 months now and still don't open up to just everyone. He is starting to get a little closer to grandparents and other close relatives. I think all you can do is be patient.
I have two children who simply prefer mom and dad and two others who are social butterflies. Just don't pressure them to go to people to whom they might not want to. (as much as you can)
Good Luck