N.D.
Setting boundaries at this age is more than appropriate. Letting him know that experimenting or "playing" with his body is okay but that those are things that are meant for doing alone, in the bedroom. It is important that you use language appropriate to his comprehension and that you re-direct as often as it takes, each and every time. Once he puts his hands in his pants, tell him that he either needs to stop or he needs to go spend some time in his room will set this boundary. I am guessing that this will nip it in the bud and he won't be so focused on it anymore. This will at least buy you some time until the teenage years. :) Just kidding. Boys and their "wee wees", they go hand and hand...literally.
Hope this helps. Good luck.
N.