D..
Have you asked your pediatrician? That should be your first stop. If you are not comfortable with his or her answer, then ask to be referred to a pediatric nutritionist.
You need to have some professional guidance and not just look for answers on the internet. It IS important to get fat for her brain development and she must have calcium to build the bones. It does not appear that you actually know if she has a dairy intolerance since you said that YOU developed one after birth (unless you mistyped here.) I don't know if you intend on keeping ALL dairy from her (that would be really challenging), but you must know with help from a professional what she has to eat in lieu of it in order to get what she needs for her brain to develop normally.
I urge you to get REAL professional guidance instead of listening to "others". (Though, yes, the "others" are right about the fat in the milk being so important.)
Dawn