Don't allow tantrums. Don't feel bad for disciplining that. It's very important he doesn't learn to get his way by acting out and ALSO that acting out isn't allowed, so just ignoring the tantrums isn't enough So depending on what you mean by "we get irritated so he gets in trouble", it's good you are addressing it. Getting irritated simply means it's getting too far though. Set the rule, give him a time expectation and a clear activity (play doh on a clear table, train tracks on a clear table, a (gasp) TV show, pick ONE thing, but not just a room full of toys, that's too distracting and nothing will interest him) give a calm warning to keep at it, and enforce calmly right away. He'll learn faster this way. At this age, you should enforce 20 minute blocks of time, then go give some attention for a few minutes, then possibly another 20 minutes and so on. He won't stay occupied for a whole hour or anything unless it's a movie. Be sure to go over and praise and hug him while he IS occupying himself, and enforce if he starts to come demand you right away.
Of course continue to engage him plenty at other times, but he is at the age where he wants attention ALL THE TIME, and it is important to teach him he needs to play alone at times, and you do need to enforce it since anything less that ALL THE TIME will not be acceptable to him. That's natural. It's your job to lead. It's good practice for him and he's not too young. I had to work PT from home with my husband gone all the time when my oldest (5) was 3. I had to give her designated activities and time slots to occupy herself. It was a battle at first, but she learned and is the better for it. Now all 3 of mine, 5, 3 and 2 can leave us alone to get things done when we need them to.
Don't feel bad. Parenting wasn't meant to be easy.