As a single mother to a daughter who is also adopted and has no close male family, it is the same with girls. You'd be facing this same situation if you'd adopted a girl rather than a boy. It is totally normal.
Men represent FUN adults for them...they aren't disciplining them, they aren't busy providing care, etc. They are adults that they just plain get to have fun with, and women are the ones who do take care of everything else. With my daugher, it's always worst in June as Father's Day approaches, as daycare focuses on dads.
You need to get to know more men for your son to interact with on a more frequent basis...church, neighbors, friends' dads, etc. My daughter adores my male neighbors, her friends' fathers, and my male friends and coworkers. It gives her a healthy interaction with them, and it takes away some of the novelty/mystery.
I would not date if not interested in that right now or as a way to introduce men into your son's life. Dating will take time away from your son that he will miss from being around you, and a relationship will divide your attention even more. You're already juggling a lot on your plate, and you don't need to add something else that you aren't really interested in doing. Besides all that, unless you find a man that you are planning to marry, you wouldn't want to introduce your son to men you're dating and then have them exit his life. That'd be more detrimental than beneficial, IMO. Plus, a relationship should be built first upon your feelings for someone else more than your child falling in love with that person. When I consider dating, I have those same issues....I don't have time to devote to building a relationship so I really have time to date, and I don't plan to introduce her to any men I'd be dating until a future together appeared imminent.
By the way, in some ways, it will get harder before it gets easier. Just wait until you son tells you, "so and so says I don't have a daddy...," we had that at age 3. Kids say it as they start to notice differences, not necessarily even to be mean. I always tell my daughter, "That's true. God makes all kinds of families which is really great! He made our family with Mommy and you, and we have lots of love. And, you have a Grandma and cousin so-and-so and uncle so-and-so, etc."
Don't play the "what-if" game or feel guilty about other family options. You have no way of knowing what other family your son may have been placed into, and it really isn't the important factor right now. He's with you . You love him, and you are providing the best life you can for him.