Hmmmm. We have some neighborhood kids who are not "the best".. They're kind of scrappy, and they're a bit older than my son, but I believe kids need "alone time" to work out their own macho developments, so I try not to be too micro-managing about it. So far they haven't "targeted" my son so much as you're describing though. I have had to scold them ALL (including my son) for some mischief, and I do hear some "not so nice" stuff going on in the yard, but again, I stay out of it. My son has occasionally cried, gotten hurt, been picked on, but surprisingly, he doesn't want to come in from playing and he doesn't want me to interfere-even when a kid threw something at him and he needed stitches! (I did talk to the kid but my son was a trooper!) When I was young, parents never intervened on wild free play and kids worked everything out. Either by adjusting, or fighting, or not being friends anymore or whatever. I come from the "be home by dark" generation. I commend you for standing back as much as you have.
This kid's behavior is pretty awful though. It seems to me that you need to "get through" to his parents somehow, and if you can't then he has to be banished from your yard. It's good that you did not attack this kid to the mom, but you may not have been clear enough. Maybe one more talk with a, "I'm not mad, and I want to work this out, perhaps I wasn't forceful enough last time, but your son is doing these______specific things that are over the limit of what I can handle. Do you think there is any way to curb this so we can still let them play together?" That way, MAYBE they'll be firmer with him, or maybe THEY'LL keep him away more often so it's not all on you. Right now you're the only one feeling there's an issue.
It must be EXHAUSTING to have to listen to every exchange etc, you don't deserve that kind of stress.
If your child is not allowed to play with him, then he cannot be allowed in front of your house. If you're not willing to take things that far in the neighborhood, then you probably do need to cave and let your son play with everyone. This situation may improve over time as your son gets bigger, and it may toughen up your son, or it may end in an even bigger blow up, but at least then, you will feel comfortable completely banishing the kid. At this point, it's sort of an odd limbo...The kid was told to go away, but he kept coming back! You told him "no" but your husband let him play! You're trying your best to be assertive, but so far it's not enough to succeed. You need a sit down with your husband and the parents to try again to be heard. Whatever you decide has to be HEEDED! If the rule is that he cannot come over, then his parents need to make sure he does not bang on the door, and you (AND YOUR HUSBAND HELLO!!) should take it up with them every time he does. I don't like how everyone is trampling you!