V.W.
The only thing that comes to mind is that he'll (all of you) just have to suffer through it one time. Maybe he won't sleep much. Probably it will be a horrible event. But, the next time shouldn't be as bad. It seems to me that it is just another version of "sleep training". He has to experience sleeping there overnight to understand that he CAN do so and that nothing terrible happens. And then the anxiety will fade a little more each time.
If he doesn't have a "lovey" that he sleeps with regularly, that would be priority #1. Get him one, and let him sleep with it every night. Then when he goes to his dad's the lovey goes too. You could also let him take his pillow from home, or if he has a favorite blanket, that might help, too. Items that help him feel comforted and allow him to still feel connected to "home". That he is not just "displaced"... his stuff can attach him to "home". Just be sure that if he takes items, that his dad understands the importance of the items being returned WITH your son, so that he can take them back and forth to connect the two places. If he just takes them there and leaves them until the next visit, he will not form the same attachment.
I know it is heart-wrenching, but his dad calling you and then letting him come back "home", only reinforces to your son that if he is upset, he needs to be at home with you. That if he gets upset enough, he'll get what he wants. It's tough. But it is much like other things that we have to tell them "no" about. No one wants him to be traumatized, but it's his DAD. Not some stranger! Dad needs to tell him "sorry, no" . Then give him tons of love and attention first thing the next morning, so that all the fuss is "gotten over" early the next day. The next time should be easier. Dad bringing him home to you b/c he is upset isn't helping him learn what is going to happen.
I'm not going to assume that Dad is just doing the "fun" stuff then handing him back for the "tough" stuff, but Dad DOES need to learn how to handle these situations. Not pass him to you to sort it all out.
Just my thoughts.