I wouldn't be surprised if it hurts his ears when he swallows.
At least stop giving him milk (and preferrably all milk products). It's been cited as the number one cause of ear infections in children. You don't have anything to lose by stopping it for a while. I wouldn't be surprised if he doesn't stop getting ear infections if you stop using it as long as you can. I had so many ear infections as a child that I have scar tissues in my ears from it. I researched it a great deal, hoping to avoid it with my daughter. She's five and has had two ear infections her whole life, and both because of either the flu or a cold. Only two!
Worried about calcium? Try to get him to eat as many green veggies as possible. The dark green ones are typically higher in calcium and other nutrients than milk and allow you to absorb calcium better than milk does.
You can also put hot water on a washrag, wring it out, and place it up to his ear (not while it's too hot, of course). The warmth will help with the circulation. Some people use a warm blow dryer on a low setting. I like the washrag idea better.
I think the main thing is to focus on trying to reduce the pain in his ears. I bet he'd drink and eat if that got better.
Good luck.