I totally get why you would be upset. That's an awful thing to say. Is he a good guy in general? If he is a good man, but said something mean, that's one thing. If he is sort of awful, and this was the straw that broke the camel's back, that's another. If he's not a good guy, start pulling back a bit.
But assuming he is mostly a good guy, I would talk to him- with hubby, or just have hubby do it. Tell him that what he said upset you both and the kids, and you think they deserve an apology. And to please watch what he says going forward.
Also, if the girls were bad for him, they may want to apologize for their behavior. I would explain to them that their behavior was not acceptable, but NOTHING they could have done deserves comments like that. And that sometimes older people can be cranky like that. Again, it doesn't excuse his behavior, but it explains it.
My dad is the cranky one. He has never said anything mean like that, but is a curmudgeon for sure. He is fussy and tends to yell a lot. Mostly, he'll yell at the adults about the kids, he tends not to yell at the kids. We've done the same thing, explain that the kids did nothing and could do nothing to deserve the crankiness, but that's how grandpa is and he still loves them.
When my sister had her first baby, they were with her in-laws. The baby was learning to crawl but slow with it. My sister's FIL was trying to get her to crawl over to a stuffed animal and started saying "come on dummy" repeatedly. My sister was horrified. Her and my BIL had to confront him and explain why this was not ok. He has tried harder now. As a side note, this was how this guy talked, so it was just a habit. But the result was that my BIL is a jackass as well, so just be glad that your husband didn't 'catch' this attitude! :-)
Finally, if this continues, or if you kids are too much to handle, don't keep asking him to babysit. But don't cut him out of their lives. From what you describe, doesn't seem necessary .