What an unpleasant situation! I think you are absolutely on track to decide that neither you nor your daughter need to be exposed to his nasty behavior. His behavior is completely unacceptable in a 9 year old--even in a 4 year old, it would be problematic. So unfortunate that the child lives nearby, as it makes disengaging more difficult.
One possible way to set up for the break is to allow one more 'play' session between them. When RudeBoy knocks on the door or comes to your yard when you are outside with your daughter, ask him to have a parent come over with him before it starts. Then, you set up the conditions for playing (no yelling, no name-calling, no rudeness in your presence, respecting you as the rule maker). Tell him that if he yells, calls you daughter a name, insults her, talks rudely to anyone, or refuses to listen to you, he will need to go home. Decide if you will give him one warning or if it's one strike, he's out, and tell him and his parent that at the start. You want the parent there to make sure s/he knows that if he gets sent home, that is why and that they need to be there to receive him. Then, let the playdate begin. Who knows, maybe he will behave? Probably not, and when he slips, tell him 'I am sad about this, RB, however we cannot have a person in the house when s/he insults us/yells/disrespects us. You need to leave now. I will walk you home now.' Deliver him home and explain what happened to the parent.
Maybe if he faces some real, swift consequences for his behavior, he might start behaving differently with you all. Or if you have tried something similar in the past with no result, you can just not let him in the house or the yard. You can say to him 'RB, [daughter] is not available to play with you now' and shut the door. If it makes you feel better, you can remind him that the last times he played with her, he was mean to you and to her, so you are taking a break from playdates with him. Maybe he will change, though I don't think that 9-year old boys particularly enjoy playing with girls at all, so he probably won't come around much anyhow.
You can tell the parents why you are sending him away too, if you would feel better by being up front and that it's more respectful. All you need to say is that the kids aren't playing well together and you think they need a break from each other. It shouldn't come out of the blue for them.
Good luck!