B.F.
Yes, I dealt with this. It was a bit scary. My son lived basically from milk between the ages of roughly 2 and 5, but he's always been healthy and happy. He is 9 now and still on the lower end of the growth chart, but he eats well and isn't picky at all, but still only eats when he is hungry. Often his brother asks for icecream and sweets and he is just: "no I'd rather have a glass of milk".
If the doctor says don't worry about it I think you really shouldn't. But I know how hard it is not to worry. I've been there.