If he hasn't seen his grandparents I suggest a visit with them during which one of them or both talk with him, reassuring him of their love and belief in new sitter. Let him know that they are OK.
Perhaps it would help for you and/or his grandmother spend an hour or more with him at sitter's home. This would help him feel that you know and trust her.
he is at the age separation anxiety happens and then to abruptly lose staying with grandparents and going to a stranger's home is difficult for him. He doesn't understand and doesn't have the ability to understand.
It's important for you to show confidence in the new sitter and in his ability to adjust. Show no hesitancy when you leave. Say something like, "I love you. I'll be back later" while giving him a hug and then walk out.
it might help to give him pictures of you and his grandparents to keep with him. And perhaps be sure he has one special stuffed animal/toy/blankie with him every time.