K.P.
I was waiting for you to say that yo usee them weekly... 4 times a year and they stay in a hotel? I would just grin and bear it. I would also schedule activities and things when they are in town. Invite some friends over for a BBQ, day at the beach, museum tour... even encourage them to take DD for a day so they can have some 1:1 time with her and you can have a little break from them!
He may not enjoy them, but they are still his parents and DD's grandparents. If they adore her, that's the important part. Your husband being sulky is his choice and he really needs to "choose" to perk up when they're around!