First, as long as you are breastfeeding baby does not need the cow's milk. Baby is already GETTING whole milk from mommy! (special made JUST FOR HIM, not a cow)
Second, even after he is done nursing (and I applaud you on extended nursing!!!!!) he still doesn't NEED the cow's milk. The MAIN reason for pediatricians pushing milk is because of the fat contect, which their brains DO need for proper development. However you can make sure your baby gets that fat in other forms. (whole milk yogurt, butter on scrabled eggs, avacado- super yummy!, daily dose of Cod Liver Oil - Nordic Naturals sells oils that are flavored. My daughter LOVES it.) If you do some research you can ensure your baby gets all the fats and nutrients they need WITHOUT the cow's milk.
As already mentioned, cow's milk has alot of harsh proteins that many times little bodies just can not digest. The Ultra-Pasteurization and homogenation the milk goes through basically kills any NATURAL nutrition and benefits milk had in the first place. (think about vegetables that are overcooked...they are mush with very very little nutritional value.)
Cow's milk is made for a baby cow. Think about that.
Vitamin D can be obtained through other sources as well such as Cod Liver Oil, time in the sun (without suncreen), leafy green veggies, etc....
www.nourishyourkids.com is an AWESOME site that provides moms with great ideas on nutrition for their kids. It is run by a nutritionist and pediatrician who know their stuff.
Bottom line, if you pay attention to what you are giving your kids and make a diligent effort to ensure they are getting a good balanced diet they don't need cow's milk.