Dont worry about the milk! Broccoli is high in calcium. So is spinach, and many other foods. My son does NOT like milk (and neither do his parents) and it took a long time (like 3-4 years) before he would consent to drinking flavored milk.
Don't take this to mean that dairy is bad, it is NOT. I definitely think it is overrated tho, and the expectation that every baby needs to drink 8 bottles of milk a day and drink it growing up is probably *causing* so much milk/dairy/lactose intolerance/allergies. Go easy on it and only give him as much as he wants - if that means backing off totally on offering milk bottles, so be it. It is FINE and he is not going to be ruined for not drinking milk. Eating veggies is more important than that, so keep that up! Encourage a variety of foods in a balanced diet.
Things that worked for me since my son also did not like any milk at all when young (and now only chocolate milk a few times a week, preferably hot chocolate!) was getting him "hooked" on yogurt. We also enjoy cheese sticks and when buying OJ, I bought the one with extra calcium. Make smoothies and such with milk in them. Add a bit of dry powdered milk when making pancakes or other homemade goodies.