Well, been there, done that. I can tell you from experience that the 2 min phone call isn't going to "cause damage." In fact, you might want to consider calling your ex every night at say, 8 p.m. and putting the boy on the phone to have nightly conversations with daddy. That's what I did, until the oaf figured out that he, too, could pick up the phone and call the kids on his non-custodial nights. Now, its a part of our routine. And, I do the same thing when the kids are with him.
Boys need a father (or, father figure). Eventually, he may come around and do things with the boy. I've seen this time and again that when the kids are little, men just don't know what to DO with them. Once they get old enough to do stuff, then the game changes. Actually, for the better.
My ex has really come a long way as a father. My boys will have their own relationship with him that is all their's. And, saddly, they have an alternate universe with me. Amazingly, they have gotten accustomed to 2 households, 2 ways of doing everything. Like a split reality. They're 19, 12 and 9 and we've been a broken family for 5 years now. Not exactly my choice, and lemons, lemonade, you know the drill.
With my oldest, and product of my first marriage, I fired the husband and kept the kid. We split up when I was pregnant after a 10 year relationship/marriage. I used to allow any access he wanted with the boy, as long as 1) he never drove a car with the kid; and 2) his parents, with whom I had an excellent relationship, supervised ALL visits. Usually, that daddy just showed up late, hung-over and seeking attention. He barely interacted with the kid at all, leaving it all to the bimbo he ditched me for and his parents. My son was my mother-in-law's only blood granchild, so she was very good to him (despite the fact that she never stood up to her dirtbag son and insiste that he grow up and 'do the right thing'....grrr...we had to agree to not ever talk about it...which made it bearable.)
Here's the thing someone wise told me: If you keep the kid away from his father, the kid will end up hating YOU. If you put down, disparage or say anything bad about dad, then the kid will consciously AND UNconsciously want to DEFEND him, because he is, after all, the result of half of his efforts. Supposedly, its a 'natural inclination' for kids to want to defend their dads. So, tread lightly there. Even if you're talking about it in the next room and you think the kid can't hear you or understand what you're saying. They do and they can. Have those conversations with girlfriends, your mother, etc., when the kid is not on property, so-to-speak.
With ex #1, I was terrified to let my firstborn go with the nutbag dad, so I orchestrated visits that made ME feel comfortable.
I forced the issue with ex #2 and put the kids on the phone every night, and it blossomed from there. As much of a jerk as he was, its worked out well re the kids. Sometimes there comes a moment when a man falls in love with his child. If you're lucky, it happens when they're an infant. For some, it happens later. Maybe you can make that happen by orchestrating nightly contact. That's what worked for me.
Best of luck to you,
C.