Nope. No red flag. I think you are over-reacting a bit. You are one of many patients. You can't expect your doc, or any doc for that matter, to have a super human memory. You visit once a month. And even a midwife sees many patients per month. You can't expect them to remember everything.
He sent you for bloodwork which was exactly what he should have done. That's the logical first step in solving a medical problem. Your bloodwork can tell him a lot. And it did exactly what he wanted it to do-told him you are mildly anemic. Heart pounding and weakness are so common during pregnancy and can be a huge # of things.
And the last thing a doctor wants to do is give a pregnant woman a laundry list of things that *may or may not* be wrong with her.
There is also a burden of care of the patient that so many of us patients forget. If you knew you were supposed to be taking iron and weren't, you should have brought that up. It may have been a couple of months since you told him you were off the iron right? You sound well educated and proactive so use those skills to help your doc provide you with the best care possible.
Best to you and the little one on the way!