N., some of the responses you've gotten so far are spot on. He's a baby and still discovering his world. Experimenting with the sounds and echoes is normal and he should be able to enjoy it and explore it. But you do want to make sure that he understands the difference between appropriate and inappropriate behavior in various places and situations. At 2 1/2 he is old enough to understand the difference between right and wrong but, when you talk to him about it, it has to be in terms he can understand.
I've never been big on rewarding little ones with sweets and toys for proper behavior. It teaches them a monetary sense of self-worth and it's not really necessary. At that age (and for the rest of their lives, really) verbal praise and knowing you are proud of them is the best reward they can get. Then, when you take him to the park or zoo on the weekend, it is not tied to a particular event. It's just because, "You've been such a good boy this week and Mommy is so proud of you. I think we should go someplace fun. Do you want to go to ... today?"
I am adamantly opposed to any kind of corporal punishment unless a child is doing something that may endanger someone. Then a swat on a well padded butt is not out of line. Spanking children and pinching them and the like merely teach them that violence is acceptable behavior. That, of course, does not mean that you should "forgive and forget" inappropriate behavior. When they do something wrong, they need to know, in no uncertain terms, that they have done so. They have betrayed your trust and disappointed you. If you are in the grocery store and, after eliciting a promise of good behavior and using only his quiet "indoor" voice he continues to act out, don't be surprised if you have to remind him a couple of times. But, if, after a few reminders, he appears not to be taking his promise seriously, you have two issues to deal with. His immediate behavior and his learning about a sense of honor. It may be inconvenient for you for a while but let him know that if he violates his promise of good behavior again during the outing, you will have to leave. Then, if he starts shouting again, take your cart of groceries to the service desk, let them know that you have to leave and, if they will, have them hold your shopping until you can return without your toddler (with the exception of frozen foods, of course.) Pretty soon, two and a half will figure out the cause and effect and grow to understand that his behavior will determine if he gets to stay or not.
Now, as far as people glaring at you. Someday, what other people think will be critically important to your son's success in the world. He will, eventually, need to learn to respect other people's feelings and opinions. And even now is not too early to start teaching him that. But he is still just a baby and anyone who doesn't understand that has a small heart. Don't worry about them. If not your ds they would find something else to grumble about. So enjoy your son and don't worry about the momentary ire of a stranger.
And remember, follow your love for him. Be free in giving love to him. Never be afraid to tell him you love him. And, though in years to come he may get angry and shout things to break your heart, he will still love you and feel secure in your love. And that is what will give him the strength and courage to say things he doesn't mean. He will always know you love him.
Raising kids is never easy. I don't think it is supposed to be. (The little ones aren't the only ones learning!) The good times will always outweigh the bad times. So fasten your seatbelt and enjoy the ride.
... or you could just borrow Gretchen's duct tape!