L.M.
I really think you should tell him. If it's too difficult for you, maybe hubby can speak with him. Remember there are tons of confidentially laws and rules so he won't tell anyone else. Tell him you don't want it noted in your file.
Another thought, maybe you might want to consider having a nurse present during the exam. Maybe hubby can stay with you during the exam. Would you be more comfortable switching to a female doctor?