I have a 4 y/o son who recently went through something similar so I can definitely relate! I really recommend the book _Your Four Year Old_ by Ames and Ing (I believe) and the rest of the series for all ages. It tells you what's normal for the age and what works best to deal with it. I don't agree with 100% of it but it's invaluable to read what's typical (and all the typical rotten stuff that you've been spared! LOL), plus lots of creative strategies to make life easier.
Be sure to model how to handle your own stress and anger in a healthy way too. Often the only things we teach by example for stress is that we hit or yell (or push our stress down inside), and so the kids don't know what to do with their own extreme feelings. I know you said you don't hit him but what do you do when you're angry? He needs to see how to let go of his emotions in a healthy way so you might think through what you'd like to do about your own stress. Some stuff that I do is exercise, go on "angry walks" (I put little ones in the stroller and just walk and walk until I feel better), write, vent to friends, clean like mad and do art.
It sounds like you and your husband are doing a wonderful job with him, and your patience will pay off with a sensitive, sweet kid. All four of my kids are very different from each other but all of them act best when they feel best, so I'm guessing that he's feeling scared or threatened or insecure about something. Be firm about what is not allowed but also up the quality time together. Give him lots of hugs and smiles and time with you. The good news is that in my experience, the willful part of this age is short lived! Good luck!