I'm sorry this is happening because I agree with others here that this is a red flag, maybe a small one at this moment, but a warning to you nevertheless.
One line in your post jumps out at me: "According to him I have no right to tell him how to spend 'his money.'"
Could you please do a "so what happened' update and let us know: What was said between you and him that he responded like that?
Did he make some specific expenditure that you questioned? Does he have a pattern of spending money on things that you feel are not necessities? It would help us here to know how this conversation came up between you and him. Maybe he spent money on something you felt was not in the budget, or was frivolous, or just for him alone, when you yourself don't spend anything on extras.
If this subject came up over a single expenditure he made, and in the heat of the moment, he got angry and defensive and made a comment about "my money" -- that could be worked out. He might have blurted it and not really meant it; you won't know if you and he do not discuss it. Say he came home with something you felt was just for him and kind of silly and frivolous; you said, "Hey, you bought X but I don't even get manicures--and we need to set aside $ this month for the kids' college fund...." and then he got defensive. Was that the case? If it's a one-time thing like that, talk to him.
If there is a pattern, or he has said this more than once, and especially if he keeps a tight rein on the money and you are subject to his rules (or his whims) about what YOU can have or spend -- then there is a bigger and more serious issue and I would definitely seek couples counseling as others suggested. Does he dole out an "allowance" to you and nag at you if you need more and ask for it? Does he keep hold of the checkbook, or deny you a credit card, or haggle with you over your credit card charges when you use your card? Those are all big red flags that he is controlling you by controlling access to money, and if he does those things, you need more serious help and counseling is a non-negotiable requirement.
Only you know if the situation is one where he blurted something once, or just gets defensive at times maybe out of guilt over spending on something for himself -- or if the situation is one where he keeps the money under his control and questions your expenditures. If it's the latter, that is a sign of a controlling nature, and that is a relationship problem, not just a money problem.
In our household, my husband is the sole breadwinner but has never, ever thrown that in my face in 22 years of marriage. Your husband is doing it after only three years. He may be feeling that your large difference in age gives him a "right" to be the boss financially or whatever. But as you can see from posts here -- your idea that all money is "our" money IS the way that healthily functioning couples look at finances.