I don't feel like most of the above posts....I don't have a clue who you should contact, I'd start surfing the county website for info, but here are my thoughts on this, for what it's worth.
My guess is he's never given you a penny for your kids anyway, so you'd have to fight way too much to even get a penny from him. And he will ALWAYS have visitation rights if he went to get them someday, even out of spite just b/c he wanted to mess w/you or whatever reason, and then you'd have to let your kids go w/someone they don't even know! I am NOT for that at all.
My oldest daughter is from my ex, we were never married, it's a different situation b/c he wanted and got visitation and has always been there for her, BUT if he wasn't and wanted to give up his rights, I would have let him. I don't want my daughter to be forced into his life, it has to come from him. I got remarried when she was 3-1/2, and if her father hadn't have been there, my husband would have LOVED to adopt her and give her his last name. He still would even though her father is an active part of her life! Of course, he wouldn't allow it. But I'd figure out how and get him legally out of the picture asap if that's what he wants. You don't need his support anyway obviously and it's not worth getting in my opinion if it means that someday he can just waltz in and demand to see them and you'd have to let them go.
Good luck in making the right decision, this is an awful situation for them and you, but they will know when they are older that it was him that didn't want to have anything to do w/them, not you. It also may be difficult b/c I don't know if the court will just let him give them up just like that w/o making him be responsible for them but good luck. How old are they?