I can understand how this can be stressful for both you & the baby.
This happened with my second child, too...except that my husband discovered he was the only one who could rock him to sleep...and so did that as often as possible. See if your husband can find something that works and then do that often.
Are you nursing? I think that may affect the baby, too - I know my son stopped crying instantly when I picked him up and I think it was largely me being the mom and source of food (not to sound cynical but i think this is such a large part of a newborn's life that it has to have an impact). If you are nursing, maybe your husband can give a bottle of breast milk every so often so that the baby can be comforted with that.
Also, have you tried swaddling the baby? Once my son was swaddled and feeling warm and snug, it was easier for everyone to hold him. my husband quickly became a master 'swaddler'!
I did feel like i couldn't go anywhere without my son crying, so I had to get over the guilt. Couldn't be with him 24/7. Sometimes, we just had to let him cry it out with dad, grandparents, babysitter...
Rest assured it won't last forever. And I don't think its a statement about how the baby feels about his dad. My son is now almost 12 and he is very cuddly and affectionate with everyone and very close to his dad.