If your husband and sister-in-law are willing and you feel your son would be fine, I say let him go. It will be a fun 'big boy' event for him to go and a great chance for him to spend time with his extended family.
With regards to the nanny issue, if your sister-in-law chooses to have a nanny for her own children, it is not your place to tell her she HAS to watch your son, instead of the nanny. You can choose whether or not to allow your son to go, but you can't tell her how to run her household. It sounds like you have a good relationship with your sister-in-law and that you trust her judgement with the grandparents situation, so you should also trust her judgement in selecting a wonderful nanny who takes good care of her children and will take the same good care of your son. My sister worked as a nanny for many years and in one of the families she worked for, the mom was a stay at home mom. They had a great relationship and the family still keeps in touch and sends Christmas cards now 14 years later.
As far as the grandparent issue goes, I have dealt with similar situations with older family members. At your son's age, I don't think you have anything to worry about. He'll likely just ignore any comments they make. And your sister- in-law seems aware of the potential situation and has probably dealt with it with her own children, so I'm sure she'll be able to handle the grandparents. Don't worry if they try to make him eat foods he doesn't like, as long as it's not a food allergy, it won't hurt him and he likely won't even remember. My grandparents do the same thing with my kids (7 & 3) and make them eat everything on their plate (which I rarely do) and they just know that's the way great-papa is. Kids are very adaptable, more so than us adults.
I'm sure you'll miss your son while he's gone, but trust that this will be a great experience for him and a chance to spend time with and get to know his extended family, "warts and all" as they say. It was hard when I sent my daughter on her first trip away with grandma. She was about your son's age. But I can tell you from experience, it will be harder on you than on him.