L.
My mom has dealt with the same grief you describe. She was never close to her parents and has issues with them being missionaries and making that more important than family. She has always grieved that none of her children know her parents. Since we are all adults, we joke when she talks about them and say "grandpa who? do we have a grandpa?" But I always have reassured her that we don't feel a void because we don't have a relationship with them. My parents are so family oriented and keep our family close. We have been fulfilled by our immediate family and none of us are lacking anything.
That is the only thing in your control. You work on your role as a parent and keep your family close. They will not feel the grief you do for his lack of involvement. Unfortunately you can only control yourself. He has made his priorities. Focusing on how that bothers you will only affect your ability to be the mom your kids need.