You have some good advice on here. If you keep this up, you won't be happy when 3 comes around, it will be worse. This age is called terrible 2's because this is the age that they begin to test boundaries. You have now taught your son that if he throws a tantrum long enough, you will give him whatever he wants. He get more and more worked up because you are completely confusing him. You give in to him sometimes, so when you don't give in he doesn't understand. He also assumes that if he keeps up with the tantrum, you have a breaking point.
You have created more work for yourself, but you can fix this. Unfortunately, you are going to have to endure the yelling and your baby may wake up. I know it's awful, but if you keep this up you will have bigger struggles in the years to come. You will be that lady eveyone stares at in the grocery store and restaurant.
As other moms have mentioned, I put my kids in their rooms for tantrums. When they got to a point where they were getting worked up, I would calmly say to them "I understand that you are upset and if you want to yell and scream, it's ok, you just have to do it in your room. When you stop crying, you can come out and talk to me." If they did not willingly go to their room, I picked them up and put them in their room and closed the door. I never yelled back, argued, spanked or punished them. Honestly, it only took 2 or 3 times of this and it has not happened again. My kids are 6 and 3 and do not throw tantrums. To this day, if whining starts, I tell them they can do that all they want in their room and it stops.
You have to view this not as a punishment, but simply as allowing your son to have his emotions in his room. He will stop because he wants an audience and when he realizes he doesn't have one, he will have no need to throw a fit.
In public, I have only had to deal with one outburst. I was so embarassed but I just ignored it. My daughter wanted a doll and when I said "no" she had a melt down. I wanted to run out of the store, but I thought why should I leave a cart full and let her dictate my life?. I just moved fast, finished shopping, checked out and told her in the car that I was sad that she would behave like that. Because we walked out of the store and I did not cater to her, she knew that throwing a tantrum would get her no where.
On a final note, your son is also reacting to the change in his life with the new baby. He doesn't know how to express himself other than through tantrums. When he is behaving, give him lots of praise and attention. He'll realize he'd much rather have that kind of attention. When you change the baby's diaper, ask him to bring you the diaper or talk to him while you are doing it. While feeding, have your son sit next to you and talk to him. Say things to the baby like "you are so lucky to have your big brother". There are ways you can incorporate your son into the time you are spending with your newborn so that he doesn't feel like he lost his mommy. When your daughter is napping, try to give your son a little extra attention. If you are doing the dishes have him put the spoons in the dishwasher, it might take you longer to do the dishes, but you will be spending one on one time with him, keeping him busy and giving him attention.
Good luck, stop catering and yes, consistency truly is the key so you don't send mixed messages, which hurts your son more than ignoring him during a tantrum.