First of all, good for you for breaking with this "tradition" that just turned into a tradition of being miserable. Divorce makes a lot of things awkward, and then people try to figure out ways to pretend that it's not awkward, which makes it way more awkward. If your brother's girlfriend pretty much dislikes everyone, then avoid it!
I do think there are a million things that vegetarians do to make the meal wonderful. So if it were purely that problem, I'd say to be way more adventurous in creating a meal that you can all enjoy. Sometimes we get locked into Grandma's traditional green bean casserole even though everyone hates it, or cooking a whole turkey even though no one in the family eats dark meat (so we throw away the drumsticks and the carcass). So it's fine to recreate a holiday in a better setting for all of you.
That said, I think it's fine to create a family tradition that works for you. I have a few problems with other people having to work on Thanksgiving so that we can go out and do something fun - although I know there is an argument that workers who WANT to work can make double-time and restaurant wait staff can make bigger tips. It is a pretty good day for a drive because no one else is driving.
We have a bit of a fractured family - my stepdaughters have issues (they have a needy mother and her single sister, one SD is married to a criminal, and the other has an anti-social husband who only wants to do stuff with his family, etc.). We reconnected with some 3rd cousins and used to have them all come to dinner, although that has tapered off due to the death of the mom and the marriage of one son to a big family with huge dinners. So even though our group is much smaller, it's much less stressful (and expensive).
We really decided to make it about Thanksgiving - in the sense of being thankful. We have only the foods we want (although I do make my son engage in the family tradition of the kids tearing up the bread for the stuffing while I prepare the add-ins), we watch football, we watch the dog show (no idea why we love it, but we do), we take a walk, and we find a good movie on that evening. We're also big Scrabble players, and that can be a put-it-down-and-pick-it-up later kind of game. We often build a fire (my son's special project now), and we sit around with the dog and play a board game as a family. We refuse to go shopping on Black Friday and we turn it into a weekend of togetherness.
What I think it most important is that you are creating family memories for your children. If you are the sort who send a year-in-review letter with holiday cards, why not start on that? You can have one of those discussions about "what are you thankful for?" which isn't bad when kids are getting into the "what I want for Christmas or Chanukah" frame of mind (fueled by store decor that's been up since Labor Day of course). If you want to spend the weekend (although not necessarily Thanksgiving Day) getting ready for holidays, it doesn't hurt to get kids involved in sorting through decorations and checking light strings and hauling stuff out of the attic too. It gives them a sense of appreciation of all the work Mom and Dad go through - and actually may make them more thankful in the long run!