Sadly, yes, it is a very good reason not to have another child, not only for the potential child, but for the amount of stress that it will put on you, your marriage and your 3 year old. That you're even asking the questions shows what a responsible and thoughtful person you are. Clearly you're very responsible or you would not have finished college-I know how tiring finals can be, I could not imagine taking them while about to have a child! i have a feeling you know the answer, just look in your own heart and head.
I may be in the minority opinion here, but I too live in Los Angeles and know how super expensive everything is, and how rocky our economy is. Maybe your husband can find a way to contribute more moeny to the household, change jobs,get a more advanced education etc.
You dont say how old you are, but it seems like you are in your late 20's or so. Perhaps you should give yourself a chance to get a bit more financially set and then revisit the question? I had my last child at 39, so there is still time! I know you're concerned about the age difference, but life is a trade off. BTW, my oldest and youngest are 6 years apart and get along just fine. I too would have liked to have another child but the reality was that we just could not afford it. I know where you're at and it is painful. Ultimately I know I made the right choice for my mental health!
Good luck, I applaud you for thinking this thrugh and not being selfish. Women like you get ahead because they look down the line, not just at their wants, but at the long term effects of their choices.