I am so sorry this has happened. It really is an uncomfortable situation to deal with, but you have some choices.
All of us have an internal mechanism that will tell us when we are in danger or when we should be extra careful. This is a primal instinct. It is what keeps us safe. We must learn to trust this feeling.
So if you feel frightened, I would call the gentleman and let him know you will no longer be able to work for him. Tell him the reason and DO NOT let him convince you differently. If you are afraid you will back down on this, you can have your husband sit next to you while making this call or have your husband call the gentleman.
If you feel like you still could work for him but want him to stop this behavior, go with your husband and have a conversation saying that you will only continue to work for him if he is not home. That if he returns to the house. You will stop at that exact moment and leave for the day. He will still be responsible to pay you for your full time and full job.
If you feel like you would work with him in the house, tell him you will only do this if you have your assistant and he will need to increase his payment to you for $20. per hour so that you can pay your assistant. If he agrees and if you can find someone to work with you for this amount or more, charge him that price. Once again, tell him if any shenanigans happen again, you will both leave and he will be responsible to pay you AND you will not return.
What ever you decide keep in mind, you do not deserve to EVER feel uncomfortable. This is your business and believe me there are tons of people who are looking for a trustworthy housekeeper.
Put everything in writing. Make him sign it or have him respond by email so that you have documentation. Hang in there. Working for yourself is very rewarding, but when these things happen, just follow your internal instincts.
****My husband just read this response. He works for the police dept and he says you should purchase pepper spray and carry it with you to this clients house. If you get into a situation you can use this as protection.****