We have a 5.5 year old daughter and almost 5-month-old boy. Our daughter is med-high functioning autistic, so I don't know if that is a factor here...
Our daughter mostly sleeps in her own bed now, but she has slept in our bed on and off since she was a baby. (*We had a firm queen-sized mattress, and neither of us move in our sleep. Having her with us was the only way to get sleep due to her need for constant contact. This is NOT a good idea for infants, but we felt the situation warranted it at the time. Our son does NOT sleep in our bed.*) It wasn't until her baby brother was born that she began sleeping in her room on a consistent basis. She still crashes with us once in a while, and we feel that's okay. Sometimes children need to be physically close. It's reassuring to know Mommy and Daddy are there for you.
As to your husband not sleeping in the bed, that's something that hasn't happened with us, except for when I can't stand the snoring. Sometimes I have him switch rooms with our daughter if we're both exhausted and need sleep--he'll crash on her bed, and she'll sleep with me. I think this is okay as long as it's the exception. If you have a king-sized bed and neither of you are prone to tossing/turning, you should have enough room for your daughter to sleep between you. You must be careful not to place her in danger of suffocation. Remember, she's still small.
It's important to be with your husband, even if you aren't being *with* him. There's something about being physically near him that keeps up that bond. In my opinion, I guess it comes down to whether your daughter's presence in your bed is interfering with the marriage. Not meaning offense to some of the other posters, I don't think that the nightly separation will necessarily lead to the end of the marriage--there would have to be deeper issues. Even so, you need that closeness.
If it's your daughter's need to be close, invest in an air mattress that can be on the floor next to your bed. You can wean her/yourself by having her sleep/"camp" on there and then moving it further from your bed toward her own.
By the way, never, at any time, should there be sex in the same room as your child. I hope that's not what you meant, but if it is, know that it should not happen. Our son was conceived while our daughter was in her own room (probably one of the nights my husband fell asleep cuddling our girl to help her to sleep, then moved to our room).
Don't worry over much. In other cultures, families sleep together all the time. As long as nothing inappropriate is going on, then it's as innocent as it seems. Some children need that reassurance, and in most cases, that's just fine.
It sounds to me like you're a loving mother, and that your daughter is lucky to have you. :-) You'll get there with the sleeping situation.
PS--Our little girl mostly sleeps in her own room now. I think it's annoying to be woke up by her baby brother. ;-)