Whatever the grandparents, or any other relative or friend, may think about your pregnancy is pretty much beyond your control, so try not to torture yourself about it. Trying to be "responsible" for how someone else handles the truth is not an exercise in mental health. Really, the best you can do is to tell them kindly and truthfully.
If any of them express a negative thought, you can simply agree, or sympathize: "Yes, I had that same though myself when I found out I was pregnant!" or "I can sure understand how you would think that." And if the opportunity presents itself to say more about how you handled that difficult thought, great. If not, then it probably will on some future occasion, if you keep communication open.
There's nothing wrong with agreeing with them, and it can actually increase the chances of deeper, more honest, and more heartful communication on into the future. If you act hurt, or carefully avoid/deflect some people's truthful reactions, you reduce the chance of connection in the future. Everyone, you and grandparents alike, has a right to his or her genuine emotional reactions.
Of course, following those initial reactions, thoughts and judgements can arise, and this is where communication tends to get sticky. But if you can keep from becoming defensive about the situation, you'll be better able to respond with humor and grace.
So, if I were in your position, I would just deal with the reality of the pregnancy in whatever way allows you to be positive. Nobody is in total control of their future, and those who most seem to be often miss out on a lot of joy. If you're handling life, family, work, and this new development as well as you're able, it doesn't matter much what other people think, unless you allow it to rattle you or get you down. It won't change your attitude or your efforts for the better, so let it go.
My best to you.