J.D.
Completely normal! It will change over time.
You say he was around a lot of people early on and this can make things a little more intense. Babies can get sore from being passed around and may associate a group of people with this pain. They also get really tired and too much stimulation can irritate. He is just telling you he wants his comfort zone which is the people he knows daily and that is ok.