I can absolutely relate to what you are going thruough. I left my husband when my daughter was 19 months -(almost 8 months ago). At the time we left, he had never fed her a meal, given her a bath, given her a bottle, medicine etc. We were together, but he worked, drank and played some sports. He was never there for us. I had to teach him how to feed her meals to make sure she was getting enough. I was and in all respects am still the main caretaker of my daughter. We have a very strong bond and I will work very hard to keep that bond.
Anyway, he also has joint custody and gets to see her a lot - not 50/50 but I had to fight that. Please know that it is all about money. The more overnights he has with his kids, the less he has to pay with child support. It is truly sad and the kids are the victims. In a lot of cases, the novelty will wear off. It is also a way that he can hurt you because I am sure he knows that it hurts you when he has the kids. Don't give him that satisfaction. Act as though you are busy and have plans when he has the kids.
In the beginning, it was very hard for me. I cried and cried and spent a lot of time with my sister and my mom. I am a little better now. I go to the gym, walk around the mall, clean house, watch videos, go to the gym again (ha ha). It took every ounce of effort to stay busy but that is what you have to do. I also would go to Borders or Barnes and Noble and get a cup of coffee and read a book/magazine. These are all litte things but they do help.
When you get the kids back, just enjoy each moment with them. Be the best parent/role model you can be. Don't give in to them and give them everything they want just because of the situation. Discipline and structure in a child's life is a blessing for that child's future.
I wish you the best of luck. Please let me know how you are doing. Maybe we can get together for coffee sometime.