It sounds like you want confirmation that you would be in the right to break it off first, so that way your heart won't be broken. And your heart is broken because you want things to be easy and natural between your boyfriend and your son and know that this would be a hard road to go down as a new family.
I don't really know what to tell you. Maybe, instead of breaking things off preemptively, talk to your guy. If you have any chance at making things better, you'll need to be very honest with each other. Parenting is hard. We just had a tough vacation with our seven year old and to be honest-- there were moments I wanted to just walk away from my kid and not look back -- and I'm the woman who carried him, birthed him and raised him. I wonder if your boyfriend sees any hope that things will change, that parenting will become easier or only continue to be hard...
Would you both be willing to talk to a couple's counselor? Find out what it might mean for you two to be in a family relationship together, what is also going to be best and most supportive for your child and how to do this *well*. You don't have to be married to go to couple's counseling, and it can really help both of you be honest with each other, validate feelings and be vulnerable with each other. These are all good things.
So, if it were me, that's likely what I'd do-- talk further, decide if we wanted to pursue counseling together or individually or both-- and go from there. Personally, if I was with a man in this sort of situation where my child's future would be directly impacted by their presence, if he wasn't willing to access outside help at all, I'd likely end the relationship. Not because I think everyone needs counseling, but because in a situation such as that, making a new family, struggling with all of the new dynamics plus the situation you describe with your son: I would want to have that sort of professional support. That might not be a priority for you, but start with talking. Go from there.