M.,
Wow - things sound pretty tough. I think it's great that you have support from your mom, and can go there to get a break. However, I think this is more than growing pains. Sounds like your husband is feeling some stuff about his own childhood, and how his father reacted to him as a baby. I think therapy would be a great idea - you need to know how serous this problem is, and what you can reasonably expect from your husband as he works things out. Also, you need to know how willing your husband is to work on this stuff, and sort it out. If he is not, then it will NOT get any better. However, therapy for you can help you clarify your situation for yourself, and offer support in making the best decisions for your family.
Try and remember that this stuff is not personal to you - I know it affects you personally, but there is nothing you did/are doing to make him respond this way. This is his stuff, and he is the only person who can change it. However, I think it would be unwise to sweep it under the rug, and not address it. This is your family, too, and you need to make your own decisions about whether or not your needs, and the needs of your children, can be met in this situation.
Divorce does affect children, but so does exposing them to unrelenting conflict and hostility. Calling a baby atrocious for crying when he is in pain is a hostile thing to do. I am not saying your husband cannot change, but he has to choose that course, and then he has to commit to it. Your job is to protect yourself and your children from avoidable conflict and provide yourself and your children the best possible living situation. It may turn out that your husband is not willing or able to take on the responsibility for working out his issues. If he cannot, then you and your kids are really better off in a different situation.
Sorry to be so blunt - I am speaking as a single mom of a three-year old girl who has been in the center of conflict for the last two years. It is really hard to look the man you married in the eye and say "This isn't working, and we can't make it work." My ex-husband has a lot of old issues from his childhood that he cannot address - too much pain for him. He doe not see how much he is hurting our daughter with his anger and pain. It's such a difficult situation. In my experience, your best course of action is to get as much support for yourself as you can, reduce your expectations from your husband to zero for the time being, and get into therapy with someone who can really help you sort this stuff out. If he won't go with you, then go by yourself.
Wishing you the best as you deal with this difficult situation,
V.