I can be the queen of rude, let me take a shot at this. How about the next time they pop into your yard, you just look at them with a huge look of surprise and say "I'm sorry, can I help you?" This may clue them in that this is weird behavior on their part. Then if they say, uh, no we are just here to play on your stuff, then you can say, Oh, well we are just having dinner, but you can come back some other time - I will let you know when is a good time to come over here.
There probably is a cultural difference and they probably do want to be friends, probably desperately, but it's wrong that they are invading your space, but also possible that they don't realize it for whatever reason.
I would always just immediately ask them, "Hi! What can I do for you?" and if they say nothing, then say "Oh, well I was just doing.... (fill in the blank).... so if you want to come back some other time, I will let you know when is a good time to come over." This puts the power back in your hands. And it re-emphasizes the point that they need to be invited to come to your house.
I guess it may be rude, I don't know, but that would make me a little crazy, to say nothing of the liability it creates.