J.S.
I am a husband, so here would be my acceptable limit.
Anything acquired while you and he are together (MIL gifts) - ditch. You don't need his permission. You dress him and are his de facto fashion designer.
Anything that pre dates you - leave alone.
A tshirt is not just about Ex's, the way some women keep tshirts from their exes. One of my favorite hole-y shirts reminds me of college. One is from my grandpa. One is from a very rough patch in my life.
He's loyal to those shirts and have kept them around. I'd think that kind of loyalty would be something women would admire. A man that knows what he likes, AND doesn't trade it in on a younger model?
Let him keep the old stuff. :)