I think that it's not up to the kids where they will live, which is why a court would not take what a 3 and a 5 year old say into consideration. Of course Daddy played the "fun man" for 6 weeks, and of course little kids want to live where it's FUN and there are no responsibilities! We've had our daughter say she wants to live at grandma's many times. Grandma's is fun, there is candy and lots of TV and attention. What YOU know, and they don't, is it wouldn't be good for them to live there. That's why adults make the decisions.
You can't prevent daddy from being the fun man. You just can't. Fortunately you know he's going to turn you into an income source and be a poor father so you can make a sound decision.
When your kids say "I want to live with daddy!" you can just respond with "sounds like you had a good time!" I wouldn't answer the question about living with daddy with a "no, you can't" because then you might have to explain. Instead, just acknowledge that daddy is a lot of fun, and won't it be fun when they go to see him next summer? If they persist, then you can just give them a big hug and say "well, you will see daddy next summer, won't that be fun? I'm so glad you are here now!"
It's best that they stay ignorant of the reason. If you feel you need to explain a little, tell them that you and daddy share the job of raising them, and they live with you part of the year so they can go to school, and they can see daddy in the summer and have fun.
No need to say more than that. And stay strong with keeping them living with you. It's the best thing for them, even if they don't know it. (And unfortunately when they get much older they will actually see why, and that's sad!)