Hi A..
First, I just want to say I feel the most for your son Clay in this because he is not an adult and still a child and being put in the middle of something he absolutely cannot control.
Even right now he is and will be for a long time, looking to you, his parents, and even grandparents in some cases, on how to deal with life, namely, drama or confusion. And guidance, thereof... It just sounds like unfortunately a classic case where some of the adults (not you) are acting like a child, and the child is supposed to be the child, you know? And he has all those emotions to sort out and figure out what to do w/them as much as a four year old mind can. Put yourself in his shoes in other words, and I would tell this to the exgrandparent as well when it is a time she seems non-condemning and open.
And my guess is that when Clay hears and sees that adults/parents/family are not able to be pleasant w/one another or come to agreement, or even 'act right', etc. then he feels let down and probably scared and confused and doesn't know who he is supposed to lean on or trust.
As for drama and mouthing and all that goes along w/and in most any family in these situations, I would suggest you look at the big picture of you and your family or you and Clay and that no matter what, take a stand today that you won't let certain things or people or what they say control you or affect you and your family unit, in a negative way.... know how your child is supposed to be raised and stand firm in it, meaning, don't let an exgrandparent henpeck you or boss you around or get you upset b/c of some statement such as 'I guess you know I won.' I can't think of anything more childish than a statement like that.
I would steer clear of the drama as much as I could and protect Clay from that as much as I could. By the way, my kids love my husband and me dearly, but from time to time (not so much now but in the past) they too have thrown a bit of a fuss or fit about not wanting to leave 'Granny and Pop-Pops.' Or whomever's house, so some of that is just that is just being a kids or that the child just doesn't want to go and maybe even he is tired from everything!
If there is a 'next' time his Dad tries to pick him up or anyone for that matter, and he is doing the same thing, just know he'll get over it and love him through it... stay positive, talk him through it, even if he cries or screams through the whole ride home, he will get over it. He does not need to see Mom or Dad or whoever, give in and learn this as a normal way of either getting his way, or the general nature of how things work. Or are supposed to. :) I hope this helps.
And I commend you for inquiring about your child on this matter b/c it speaks of who you are as a person and that obviously you want the best for them. Remember, we are all (Mothers) that are teaching and showing our kids daily how to grow up and one day become a loving, responsible and respectible adult. I sometimes have to remind myself of that daily! :) Best Wishes and God Bless you.