So sorry about this. You just have to put your foot down and tell your parents that you know that your divorce is a loss for them since they liked you ex, but the fact is, this is your family now. They are disrespecting you, they are not considering that it makes these family get-togethers miserable and stressful for you, and they are totally ignoring the fact that it confuses the hell out of the kids (witness their behavior). The kids get used to separate parents and households, and then Grandma/Grandpa come in and pretend that this is not their reality! And maybe your ex doesn't want to spend a whole ton of time in this scenario either but doesn't want to insult them. It's up to him to make a new life with family or friends, and not up to them to fill the bill. It's one thing when grandparents want to be civil to the ex, but this is every few weeks and it's ridiculous. What if you date someone seriously or get remarried? How in the world will they handle that? Just cut off your ex entirely after they've established a pattern?
Tell them they are free to have lunch with "Jim" on their own, send him birthday cards, whatever they want, but you aren't attending. If the kids have a game, a school play, a first Communion or church pageant, a scouting event, or something else, then it's nice if everyone can come and be civil. If they can sit together, even better. If you are having a birthday part for one of the kids and you really want to include Jim, that's up to you. But none of this "big happy family" charade for your parents' birthdays, your birthday, Halloween, Memorial Day, or no occasion at all. The children are entitled to see their father in HIS life and social circle,
Tell them that you give "Jim" plenty of time with the kids during the week, and if there is a weekend when the grandparents are visiting that takes too much time away from Jim, you'll happily give him extra time during the week. (Don't get into the fact that he needs a break from the kids unless you think it will help.)
Tell them they can spend time with you and the kids, or they can spend time with Jim, but they cannot spend time with the kids and Jim together unless it's one of the occasions I mentioned above (school play or other public event). If they want to send him a birthday or Christmas card, that's up to them, but you don't want to know about it. He is your children's father, but not your husband and not their son-in-law.
I wouldn't tell them he doesn't care to join you unless that's true, but finding it out would involve your ex too much in your thoughts and plans. If you have that sort of relationship with him where you can tell him you want to break your parents of the happy family gatherings to take the pressure off everyone (him, you, the kids), then fine. Otherwise, go ahead and be confident in your decision and tell them that's how it is. They go with you and the kids and act plenty happy (with ZERO references to how they wish Jim were there!), or they have lunch with Jim alone if they are so close to him. Remind them it's been 3 years and it's time for them to get with the program.
Be strong. I know it's hard, but try to get this dealt with before Thanksgiving or any other major holiday.