M.P.
You will feel less awkward over time. Sounds like this won't last very long. Hopefully, the person doing the supervising stays out of the way. Often they will bring a book or some activity of their own and only half-way listen to the visit.
I suggest you take a game or toys with you that you know your son will enjoy and then use them as an ice breaker. If he likes to draw, drawing is a good way to get a conversation started and allows him to express his feelings too.
Good for you maintaining contact over the phone and at school!