I've gone with my almost 4-year old to doctors visits since he was lttle. I bring a small stroller that will fit in a small space (Volo) just in case, his bottle and later a straw cup with juice, a toy or car to play with. Schedule the appointment when baby's OK, either ariund naptime or when he's usually quiet. Late morning and early afternoon usually work best for us to avoid early morning and afternoon rush hour.
If you're having an exam where you can't bring your child, schedule it very early, take your child out in his PJs (still asleep if possible), a change of clothers, some food and toys, and go with your hubby in two cars. He can stay with your baby and leave for work directly from the doctors office if he's short on time. Or if he can go in later, just have him look after the baby at home.
Most moms I know do this and doctors even have toys in their offices for the kids. The only place where I haven't been able to do it is dentist's offices (it's hard to reasure your child and they're very cramped).