Very typical for kids to bond more with one parent - but this will change throughout his life and one day you will feel as left out as Dad does right now. Keep leaving so Dad and child have to get together. Also have Dad be the one to take the child to something fun (zoo, park, playground, ice cream) and put him in and out of the car seat. Sometimes a change of scenery is really key. Also putting him in the stroller so he can't see who's pushing him while he's distracted by surroundings - that's what we did when we left him with a sitter, put him in the stroller, say good bye, and have her go in one direction while we went in the other. (Learned that one from a sitter at Disney World - she took him through the grounds and we went to the monorail, and he was fine in 3 minutes.)
Try not to let it distress you - kids pick up on your change in behavior, the inflections in your voice, and so on. The more confident you and Dad are, the better. If you leave, it will be very helpful if Dad gets out a really fun toy or puzzle. I'd buy a couple of "Dad toys" and put them in the closet, to be brought out only when you leave. Dad shouldn't try to convince the child to join, just sit down in the room and play. He can say, "This is really fun, do you want to try it?" Mostly he should laugh and say other expressions of fun and joy, but not super loud in volume.
Then Dad will be the super fun guy and you will be just ordinary Mom.