How does your son feel about this? Do you think he feels rejected? Is he not getting what he needs?
The schedule is sort of irrelevant - it is what it is, and lots of families, even those that aren't blended, deal with parents working opposite shifts, etc. What's important is whether your son feels like he's getting what he needs from your boyfriend when he's around. Does he crave his attention? Does he clearly long for more from him? It might be that he's fine with the situation and that it's you who is disappointed that your boyfriend isn't meeting your expectations. That's important too, but it's important that you understand the the distinction too - are the hurt feelings coming from you, your son, or both? If it's your son as well as you who is feeling hurt, then I think you need to end this relationship. If it's just you, and you know your son is satisfied with things, then maybe you could adjust your expectations and truly be happy - but if not, move on.
My oldest son's father moved away before he was born. When I met my (now ex) husband, his daughter and my son were both 3. Everyone seemed to bond when it was just us hanging out on weekends. When we got married and moved in together two year later, and the reality of parenting a small child set in on my ex, he withdrew from my son. My son was crushed, because he finally had what he thought was going to be a "dad" and he was rejected. We had two children together in quick succession so we really worked to keep things together but finally split up a couple of years ago, after 12 years together. I regret having not ended things sooner and if we hadn't had kids together, I definitely would have shown him the door much earlier and salvaged what I could of my oldest son's childhood.
On the other hand, my step-daughter and I weren't super-close. She lived with her mom and spent every weekend with us. Where she already had a mother, and my ex really missed her and made up for lost time on the weekends, I was definitely the 3rd parent in the relationship, which was fine. When she was with us, she was spending time with her dad, her half-siblings and step-sibling, and friends. My ex would sometimes complain that I wasn't living up to the picture in his head of "girl bonding" with my SD, but she wasn't really into the things he imagined that we should have been doing together. We did do a few outings together each year without the boys, sometimes just the two of us and sometimes with other female relatives or friends, but we weren't besties or anything.
At the end of the day, I would recommend couples/family counseling. Having a 3rd party talk to everyone and look at the relationship dynamics can give you clarity that is hard to find when you're in the middle of things. I wouldn't say to just end it outright, as that would mean uprooting your son again and going through the expense and logistics of a move, etc. but don't let that hold you in a relationship that's not going to work in the long run either. Do your homework with a counselor so that you can make a truly informed decision.