L.C.
I don't blame you one bit!
Your top priority is your son's safety. If your ex is taking him to an undisclosed place and won't even answer his phone, that would scare me.
Until your son has the verbal skills to tell you what's going on when he goes with his daddy, I'd put some restrictions on their visits.
Maybe you could start meeting each other at a park or other public place. This way you know your son is in a safe environment, but he still gets to spend some time with his father.
I hope everything works out for you.