My MIL... when her Husband died, planted an Olive tree in her garden... to honor her Husband. The Olive tree is symbolic of peace. And it is a really pretty tree. She planted it, in a special place in her yard. And that is where she goes to, "see" him.
If you have a yard, you can do that perhaps.
And perhaps decorate it during the holidays.
Your Husband is still grieving. Does not look forward to that day. But that is normal.
When it it is Father's Day, I get sad too, because my Dad died several years ago. I don't have a Dad anymore.
I would see, what your Husband is wanting. Rather than, what his WHOLE family wants.
Maybe, perhaps, your Husband may only want to be by himself with you and your kids. Not others? That is also, a possibility. And is common. Maybe your Husband needs a personal way... to honor his Mom. And it can be HIS "tradition." Not it having to do with EVERYONE. I too, didn't want to be among others, while I felt sad about my late Dad. I only wanted my, nuclear family. Or myself. It is very personal.
Thus, I would see what your Husband's wishes are.
It should be him, that decides.