Don't panic Mama!
My daughter is 3 (almost 4) and we have a 2 month old nephew that we spend a lot of time around. I think it is just a phase for baby to get overwhelmed. She is seeing better and hearing more at this point so things tend to get overstimulating for a while. maybe try including your little guy in the "quiet time" activities with baby? it depends on his temperament obviously, but my daughter gets pretty sucked in when i tell her that I could "really use her help with baby." The bigger of a deal you can make it sound (while talking as calmly and quietly as possible) the more he'll get into it. Ex- "Hey little man, it's baby's bath time. You know how to take a bath huh? I bet you could help me teach baby how to take a bath!" Then have him do things like squirt the soap on her washcloth and lay out the towel. The more "quiet time" activities that you can convince him are the coolest thing ever the better.
Another thing that I've found helps is having a special thing that he gets, only if he's quiet, while baby naps. Maybe a special Music CD that mama made only for him to listen to while he colors and baby naps? Pick songs that you like too so you can sing along and teach him the words (most likely while cleaning up something, but hey it's worth a try). I know they lose patience fast at this age, drives me crazy, so it's worth taking a day to make "ben's" quiet time box. Fill it with a few different kinds of activities (solo stuff, games with mom, ect) Also, a white noise machine for where baby is sleeping might help on her end! Hope this helps??