S.F.
There's absolutely NOTHING wrong with having a life outside your family. Finding the right balance FOR YOU is the key...and that balance will change as your family grows up.
For example, from the time our children were born, my husband and I made it a point to do one thing together each week, just us. For nearly a decade, this was playing in a concert band together. As our family grew up, and the children needed us in different ways, our activities changed too. Now, with our sons at 12 and 16, we both hold 2nd jobs doing something we love (I work 8 extra hours a week at the local bookstore and my husband is a DJ for parties and weddings). We love the time away and so do the boys. It gives them a chance to show us responsibility and get a break from mom and dad.
So, I say no guilt. Happy moms (and dads) make happy families. If mom is unhappy, no one in the house is happy. Find your balance and don't let anyone talk you out of it.
Good luck.