Allowing him to take advantage of you shows disrespect. DO NOT allow it to continue. You make your appointment and if he has to find another method of child care, it's HIS problem. Let them know IMMEDIATELY that this is what you're going to do. They should understand that. If you're doing something that causes you stress, ill feelings, etc., then maybe you shouldn't be doing it. You have enough to handle on your own.
HOWEVER, if you can work this out amicably, (and there is NO valid reason you shouldn't be able to) it shouldn't be a problem. Lack of communication is one of the biggest problems in relationships. If people would communicate and handle situations AS THEY ARISE, many issues would be avoided.
Start giving him bills or invoices of the hours you've put in. Make a statement at the bottom that if not paid w/ in three days of the invoice date or whatever, then a late fee of $XX or a certain percentage will be added.
If you allow people to take advantage of you, they will. This is NOT being mean. This shows that you stand up for what is right (integrity) and at the same time, you will be avoiding future confrontations, hard feelings, etc. If they want you to take care of their child, they also need to show you some respect in taking care of your child, as well as yourself.
If you're sacrificing nap time for your son, this is the LEAST they can do. They would be paying a house keeper, baby sitter or whatever, so why allow them to take advantage of you?
In the end, even though it MAY ruffle a few feathers initially, they WILL or SHOULD respect you for it if they are mature at all. Let them put themselves in YOUR shoes.