I let my son be very silly at dinnertime. My definition of good manners does not include sitting quietly. I never want dinnertime to be a chore. Chew with your mouth closed, say please and thank you, use a napkin, be nice, those are things I expect/will expect. Sitting quietly while adults finish eating is not. Maybe when he is much older. I agree with Gailiski, if you want him at the table while you finish, it should be fun family time- otherwise what is the point of him being there? If you want peaceful dinner, let him go play. So I agree with you. If the table is for dinner, and he is done, let him be done and make bird noises in the play room. I will expect my older kids to be able to sit and wait, but I would rather they join in and have family time than sit quietly.
You should probably decide now how you want dinners in your family to go. I do think you should always start things the way you want them to be later. Otherwise it is confusing, and at what age do you start saying "No, that is not ok anymore"? Even at a young age. It is practice for when they are older even if they cant get it right now. So if you want him to stay at the table and wait quietly, you should probably start that now, even at age 3, but be prepared for a struggle. If you want him to be there but allow fun, do that. Or decide that the kids can get down after they have eaten and politely asked to be excused and the adults can finish and take their time. Also, your son should listen when Dad says something, and time out is an appropriate consequence, and distraction is a good way to avoid this. But basically I think the problem lies in your husband's expectations of a 3 yr old, and that you guys should decide now what approach you are going to take for family dinners and stick to it.