I think this situation will likely be harder on yourself and husband than the 3 year old. Others have already mentioned what you can tell them about the whats and whys, but you do need to be prepared to tell your 3 year old why you yourself may be so upset. Personally, I cry as if my very best friend has died when we have to have a pet put down or they die and have found that it is important that you don't hide your grief and share why you're sad to show your child that grief is a natural part of life. Just make sure that you don't dwell on it too much and just move forward.
On the point about whether our pets go to heaven, we're having that discussion in our house right now. We've had alot of pets over the years and I always told my son whose now 11 that we'll see them again in heaven. But, he was very sad the other nite as he said that since animals don't have souls, they don't go to heaven. That is hard for me as an adult to accept, so I think it's perfectly okay to tell a 3 year old that they go to doggie heaven and whether that's the same heaven as where we all go, no one knows, but we can always pray that it is :) I tend to over-ponder things like this, but if it is true that they don't go to heaven, it's just further justification for the importance of treating them well in their current life.