I remember feeling that way and fortunately my in-laws lived closed to us so we always had "babysitters" we could trust. However, when we moved from Illinois to Kentucky, we did not have family so I had to find reliable sitters. For that age, I would hire sitters that were older...late teens, college age. As the kids got older, then I was more comfortable with younger babysitters especially with the ones who went through the babysitter class and learned CPR.
It's very important to nurture your relationship with your husband. Do you have any family members you would trust to watch your toddler? You need to schedule a "date" on your calendar like you would any other appointment. It's that important. My husband and I make it a priority to get out at least once a month (but really like to more like twice a month). Those moments are spent being pampered on at a restaurant and having conversations that we aren't able to have in the busyness of our home. Or, just remembering why we are so much in love. Love is not just a feeling, but an action and something that constantly needs tending to and the most important relationship in my opinion (other than with your Creator) is with your spouse.
I hope you can make some time to get out! It's too important not to.