Your pediatrician is definitely the best person to ask (breastmilk affects the baby, not the mother, so your pediatrician should advise on the baby's well-being), but I believe the information you received with your first child is out of date (especially if they're older than 2 years).
The guidance from the American Academy of Pediatrics changed in 2008 regarding food allergies. Currently, the statement reads that babies shouldn't be exposed to foods if they are a choking hazard (not an issue here) or if there is a known family food allergy.
The current thought is that the longer the child is not exposed to a food, the more likely the body will react with an allergy.
Here's what their parenting website says about your nutrition while breastfeeding: http://www.healthychildren.org/English/ages-stages/baby/b...
There's much more on their site, if you enter "Breastfeeding" into the search field, you'll be able to find more information if interested.