Most boys are rough and tumble and curious.
No, I would not invite that boy over. You do not have to.
Now, besides saying no, you tell your son why.
He is 9.
He is old enough to know the truth. This is a life lesson. Do it now before he is in middle school.
Per my kids, I never sugar coat things. Since they were 2 years old, I have ALWAYS talked with them and have taught them about how to "discern" people/kids/situations, and how to choose friends. What is a good friend and what is not. So that now at their ages of 5 and 9... they understand and know... how to "choose" friends and not just be followers. And they know how to be themselves and to speak up too. If other kids are being "bad." They are very good, at choosing friends now.
The thing is to teach a child how to discern these things... so that as they get older, they know how. And because, as they get older, more "bad" influences comes up. I always explain my stream of thoughts, in these things and the why's, to my kids. So that they too, will attain "how" to think too, about social situations and about other kids or influences.
Most teachers as well, know which kids are bad influences. To keep a child "friends only in school" and not in play dates, is, incongruous.
If that were me, I would talk to my child about it. And the whole situation.
By way of association as well, with a kid like that... your child can also be or get into trouble. At school. I tell my kids that.
There are times, when a Teacher will separate 2 kids. Because of a parental request or because, the mix of the children is not good. I work at my kids' school, and have seen this happen. The Teacher will inform the adults who help at the school. BUT, sometimes 2 kids will just not listen... and they will still play together. So that scenario happens too.
The thing is: your son is 9... and it is time to talk with him, honestly and openly, about the why's of social situations like this and about "influences" that abound. He has to learn... and be taught... the skills and aptitude... about how to "choose" friends. How to choose friends. And the repercussions of it good and bad.