We have three boys - ages 6, 5 and 9 months. While we're more adjusted to it now - it was tricky for us at first.
We already had our older boys in bunk beds - so room sharing wasn't a challenge for us.
Unfortunately the world is really built for a family of 4...and so was our car, table and other things. We traded in our car for a minivan since there wasn't room for three carseats in our car. The kids already outnumbered me during the day, but for bigger outings, we evaluate each instance and "assign" kids to one of us. That way we know who is responsible for watching which child. It keeps us from ripping out our hair. Also, for some outings like the zoo, etc we sometimes bring my Mom or 14 yr old niece to help even things out.
Because our schedule is so busy, we've developed reading games for the car (we read trucks, signs, etc). I think it comes down to being creative with your time. Play dates with our neighbor are wonderful as well. The older boys each get one-on-one time with us in the evening and then the baby gets alone time during the day while the boys are at school or at a playdate across the street. I think the important part is the more children you add, the more flexible YOU need to be. We have a daily schedule that we follow and I stick to it, things happen to throw off the schedule all the time. The important thing is to roll with it and do your best to keep things moving.
Because you have less time, planning ahead for meals becomes more cructial and housework becomes more challenging. The big boys are in charge of picking up toys each night (with some help from one of us). The rest of the work is broken up and I do a few things each night along with 1-2 loads of laundry a day. By breaking it up, my house is more manageable and while never perfect, I'm not ashamed of it either. :)
It's a tough choice, but I say to go with your gut on what you think you can handle. If the hard days outnumber the great days, then I would think twice. Otherwise, there is always room for one more. :) Best of luck!