S.,
We had a very similar situation with our neighbor boy two doors down on our street (lives with Grandparents, visits parents 15 miles away one weekend a month.) Just as you described, something socially is amiss with the whole family.
Because no one had taught our neighbor boy otherwise, we had to explain that he could not come in our house or yard when no one was home or watch us through the windows when we were home (and many other things.) I felt guilty asking him to knock before entering our home or eating our food, and saying no when he constantly asked to go places with our family. Eventually we had to say that he could only play with our kids in the front yard when I was out there to supervise because there were so many problems.
My best advice is to draw some very clear boundaries. Decide how much you can handle of this family and stick to it. Explain to his Grandmother when and how often he can come over and he needs to stay at the home (no more taking him places.) If they can't respect your wishes then you need to decline the invitations altogether.
Teach your son to express his needs and feelings clearly to this boy without being cruel. "I can play with you today at recess and I want to play with _____ tomorrow. I am still your friend, but ______ needs a friend, too. I will play with you again next week." This boy will have a hard time swallowing this, but he needs to learn.
In my situation, I would sometimes become frustrated and wanted to talk negatively about this boy and his family. But I held my tongue because I would only hurt my children and thier relationship with me if they heard me talking about someone like that. The good news is another boy moved in about 1/2 mile away and they ride their bikes to each other's homes. Our neighbor also joined a sports team and is much less dependent on our family. Despite the size of your town, hopefully something similar might happen. Or maybe you could suggest to the Grandmother that the boy join a team or group to expand his social circle.
Best of luck,
S.