My son was three when we put our dog down. Unfortunately, he had to come to the vet's office with us, when we had it done. On the way to the office we explained that the dog was very sick, and he had to see the vet.
He, of course, waited in the waiting room. He asked where the dog was, and we simply told him that the dog was so sick that he died. We told him that we would all miss White Fang(our dog), but he was never going to be sick or hurt again, but be able to rest.
Most three-year-olds can understand death, on some level. They've seen roadkilled squirrels, dead bugs, etc. So, it's not a completely new concept. Chances are, your child will take it better than most of the other family members. He will probably say that he misses the dog from time to time. Just remind him that you miss the dog too. It will pass, probably faster than you think.