H.P.
The longer I go without sex, the more I don't want it. My husband and I were still newlyweds when I became primary caregiver for a terminally ill close family member. As much as I needed his support then, sex was far from my mind. I was so emotionally drained and vulnerable and hurting.... I had to travel without him for weeks at a time with my family member, and I just turned that part of me off. It took a while for me to get it back upon my return.
Recently, I've had to explain to my husabnd that I need to connect with him before he can just shove something inside me. i need to unwind from my crazy day and laugh with him a bit. That's not always reasonable.
My point is that it sounds like you have the answer to why--you are perpetually wound up and need to connect first. You two are on a cycle (neither able to give to the other until your own needs are met) and haven't figured out how to jump off of it. The blame is on both of you, and so is the responsibility to fix it. You're at the point where you might not be able to do it yourself. Are you ope to the idea of therapy? This isn't to "fix" either of you but to help you with your mutual communication breakdown.
(By the way, I don't think that his accusations are because he's doing it. It's more likely that he's protecting his ego. Certain people take this type of rejection very personally. Many people have sex without ever being vulnerable. When you have to ask for it at the risk of rejection, that is humbling, and that makes it more than uncomfortable.)