Now is definitely the time to talk about stranger danger. I know it's difficult, that you don't want to destroy your son's innocence, but when he does things like run off in the mall and says things about wanting to go to a stranger's house and see their toys, this is a catastrophe waiting to happen. My son was about the same age when I had to start doing this, and though I felt bad about it, I was very frank and told him that he could not just go up to strangers and talk to them, especially if Mommy didn't know the people either! I didn't go into great detail because at 4 a child won't understand what might happen, but I did say that some strangers were just not nice people, and that he couldn't trust everyone he saw because a bad stranger might try to hurt him. It frightened him a bit, but I personally thought that was a good thing. I did not phrase it so that he got so scared that he wouldn't talk to anyone at all, though -- you don't want to frighten your son into being scared of everyone and everything, but you do want to make him cautious and start building the foundation for street-smarts now.
I tried a harness on my son and that worked for about one trip to the mall. However, I only had one child to deal with, not three under the age of five! You might try one -- not a wrist strap that he can remove himself and which would put you back in the same predicament.
I hope you find a solution that works for you!
S. :)