J.
Personally, I am with you. I have two girls, 2 1/2 yrs and 7 mos. I don't think that babies should be far from their mamas. And, although we all have occasions that we need to be separated, I am very, very, very picky with whom, how long, etc that this occurs (we don't have family around either). I have found that this is the road less traveled and that it is challenging in terms of finding time to unwind and couple time but we have survived this far. Creativity goes a long way in finding time and in diplomatically rejecting eager helpers in this regard. We plan in house dates for when the girls are asleep, for example. I found that I had to rescript my way of thinking to fit my parenting style/choices and still achieve the same end of rest, relaxation and maintaining a marriage.
I say follow your heart. You'll know when you are ready.