Tough to answer without knowing him or you.
For our "Zero" birthdays (30, 40, 50) we've had some sort of party, usually at home but occasionally in a restaurant with a separate wing or room. We usually just do buffet appetizers that are hearty (pasta, cocktail meatballs, and a variety of side dishes and salads) all of which can be made ahead and kept on the table in sterno-fueled chafing dishes. I'm a big make-ahead person. Then I have one or 2 people on hand to help with the heating and passing of last-minute or hot things. One year, on my stepdaughter's advice, I moved the living room furniture to the perimeter and rented 3 or 4 cocktail height tables (bar height) for the newly-opened space. I put a votive candle on a little mirrored disk (bought at craft or party stores) and a small round tablecloth, and scattered that metallic confetti with the numbers on it (40, 50…). Boom - done. People mingled and chatted as in a bar, and had a place to set down their drinks while they ate from buffet plates.
We displayed photos of the birthday celebrant on poster board or foam board, but if you have a creative side (or a kid!) to do a video scrapbook that can run on endless loop on a laptop, that's fun too. We also asked guests to consider bringing something creative, and the invitations suggested things like a collage, a poem, a song, a limerick or an anecdote. There are always a few guests who can write a song parody, and then they either sang on their own with a few willing participants or they handed out song sheets so everyone could sing along to a familiar tune ("I've been working on the railroad" or "Nothing could be finer", anything people know). Then a few people would read their limericks and others would tell a funny story from childhood or the office, and that was the entertainment. Even those who didn't feel talented enough to compile something could participate in someone else's offering and enjoy.
If this doesn't appeal to you, then absolutely take a trip for just the 2 of you. If he wants a man cave, I'd leave that to him to complete on his own unless you are positive - maybe he's just talking about it but doesn't want to invest the time or money.
Whatever you don't choose for him this time you can use for yourself whenever your birthday comes around!