You have not given enough information to be able to give an answer. Is there a reason that Dad can't take them for the weekend? Have you has a discussion
with him about this or did you just mention it? I suggest that you ask him to make a schedule so the kids will feel more secure. It's important for kids to have consistency. And a schedule will help you plan better. The schedule doesn't have to be every other weekend. Is there a reason for him to have them on weekends, aside from the court order?
The goal of parenting time is to help father and children have a good relationship. I hope you're not complaining to the kids or anywhere the kids can hear you. This is very important. They are a part of their father. When we complain we reduce children's sense of belonging. We influence their attitudes towards themselves and others. The children feel less secure. It's important that they know their father loves them. Whether he visits or not is not an indication of love.
If you and your ex aren't able to work together, I urge you to get counseling for yourself to learn how to make getting along more likely.
I don't understand why going to court would be helpful. The judge cannot force him to visit. You could change the times he's to visit. I suggest he will have to agree to the change. No court is going to remove parenting/visiting time entirely. I suggest that going to court will not help in getting him to visit and is time consuming, a hassle and expensive.
Find a way for the two of you to communicate in a respectful way. Whatever agreement the two of you make together will be accepted by the court.
If your goal is to stop contact, there is no way to do that unless he's not visited for a lengthy bit of time (over a year or more)or you can prove he's unfit. Without those 2 issues, it's important for your children to see their father even if it's sporadically.