I agree with Girlie that having him involved with you (in the delivery room, for example) is entirely different from having him take responsibility for his children (both financially and in terms of visitation).
He's immature, obviously, and he's verbally/emotionally abusive. Some women can get past the cheating part if the man is truly remorseful, but your ex is not only continuing with this behavior, he's rubbing your nose in it by talking about how great his gf is (vs. criticizing you). So he's not going to change, at least not for many years, and there is nothing you could have done to change this.
But why isn't he paying you child support for both the 2 year old and the baby to come? No, you don't "need" him to do that and you are financially able to handle it, but why should he be off the hook? And, if he's going to see your older child, how do you plan to explain to both children that one sees Dad and the other doesn't? I don't think that visitation is a reward for child support, but I don't think it's something you can hold over his head just because you are (justifiably) mad at him. So picking up the 2 year old and taking her for the day is one thing, but "hanging around" is NOT! He does not need to be in your home - just pick her up at the door, and he can provide a crib and food and diapers and some clothes and toys for her so you aren't jumping through hoops packing everything all the time. Obviously he won't take the baby for a while especially if you are nursing, so work that out with a lawyer. But he's not entitled to sit around your house to bond with the baby either. Maybe you can find a neutral party to help out for an hour or so.
I'd get a court order for limited (and supervised, if necessary) visitation and regular child support. His paycheck can be garnished and the money deducted by the state if it's going to be a problem getting him to pay on time. And child support doesn't "entitle" someone to visitation. They are separate issues.
In any case, seeing his children and being responsible for them has nothing to do with getting your family back together.
Get a good lawyer and get this figured out before the baby comes. You need to be protected financially, and that includes medical benefits for the kids (and maybe you), plans for a savings account for things like braces and college, and much more.