To me, it sounds like the parents were being gracious to extend the invitation but probably have no expectations of mandatory attendance. I'd politely decline saying that your schedule is already busy with other commitments (which it very well may be with girls the ages of your daughters).
However, as a Mom, I know I sometimes feel obligated to invite people so their feelings are not hurt by not receiving an invitation - my feelings are never hurt when they can not attend.
My guess is that there's no expectation of a gift, but perhaps a nice thing to do would be to have a card (signed by you and your daughters) and maybe a little recognition of his birthday when you're caring for him.
Good luck! If you're really uncomfortable, you can always stop by the party with your girls long enough to drop off a card/small gift and say that you can't stay because of other obligations.