D.D.
You might not want to hear it but an 8 yr old really doesn't have a lot in common with a 4 yr old. Even a 6 yr old is pushing it. You can't expect your nephews to put their lives on hold just because you and your children are there to watch them. It's their house and they are use to their rouine.
I'd say that you need to find a balance. Have a set amount of time where their friend can come over to play; like 2 hrs. Have a set time when they can go over to their friend's house. Your son should not be going along on their play dates at someone else's house.
As far as your dream of moving into town and having everyone play together happily every after, well that may or may not happen. A lot of getting along with people hinges on personality. I grew up with 16 cousins and I was only really friends with 2.