I can't really give much advice here, but I can share with you my DH's story, which is similar to yours.
DH's mom got pregnant when she was 19. Married the dad. Split up 2 years later. Dad left the scene, never to be seen or heard from again. Growing up, DH never had any contact from bio-dad. When DH turned 18 and still had never heard from him, he decided that was it - he wasn't going to care anymore.
About 10 - 11 years ago, DH ended up somehow being contacted by a half-sister he had never met before - a half-sister from his bio-dad (I had not met him yet so I am unclear on the details). They kept in touch for a short while but then lost contact with each other. Around that same time, DH was working at car dealership and met a customer that happened to have the same last name. Turned out they had the same dad. Freaky!
Just a month ago, I ended being found on FB by the half-sister - she had been trying to track DH down again and found me on FB (DH does not do FB). We wrote back and forth some (we live in a different state now) and she helped fill in a lot of the gaps. Turns out DH's bio-dad fathered something like 10 kids with 5 different women, and my DH was NOT the oldest (he always thought he was, turns out the dad fathered a girl with someone else before that, had nothing to do with her), then he had DH with DH's mother, then went on to have 8 more kids with 3 more woman. The half-sister also said that the dad is an alcoholic and while he was more a part of her life than DH's, he wasn't a big part of it. She said he just does not communicate or try to have a relationship with anyone - his only relationship is with the bottle. He's not necessarily abusive, he just is not engaged. I told DH about being contacted by his half-sister on FB and we talked about it. Apparently his bio-dad would probably talk to him if DH initiated it, but he would not reach out himself. And DH is still not interested - he's been dealing with it the best way he knows how and isn't about to start all over again. I told DH what his half-sister told me - as much as it may hurt, and it sucks, and it is not right to treat your children this way, it doesn't sound like he is missing much. Turns out also that DH's bio-dad was the "black sheep" of the family and the rest of the family (grandparents, etc.) wanted nothing to do with him or the children he sired all over the place.
If you feel that the relationships you could have with your father's family will be positive ones, than by all means, do what you feel is right. If they bring you more grief and pain than joy, you would be wise to distance yourself. I've never believed in having toxic people in my life that would just drag me down. But sometimes you need to be the bigger person and take the first step.