I know for many it's tempting since this is a very affordable family vacation. But I say, go with your gut, not anyone elses advice on this one. Only you know your family dynamics. If you know the ride to site will be crazy and actual camping will be a whine-fest, you might want to take a raincheck.
My husband and I are avid outdoor people, but when the kids came along quickly realized it was going to be a long time before we felt it would be doable, let alone safe for us to camp with our little ones. People tend to forget, campsites are full of many hidden dangers, ranging from strangers to seemingly friendly but dangerous wildlife. Things can really go from nice to nasty if you don't go into "any" camping situation with a hardcore safety plan. Danger isn't only reserved for those braving a climbing expedition on on K-2. Kids go missing and tragedy strikes more often than people realize at park reserves and mobile home/camp sites all the time.
If your kids are anything like ours, and they dart off, get into everything, pull-away, get easily overwhelmed or over stimulated in new surroundings, or have allergies etc. you might want to wait. You won't have much fun let alone time to spend with the grandparents, since you'll be worrying about keeping them safe and chasing them down the whole time.
Another thing to consider is weather and your surroundings where you will be camping. Will there be open fire pits, others (strangers not with your party) at nearby sites, open water, pools, or heavily wooded trails where they could get lost?
I say think about it carefully. There is plenty of time for them to enjoy a full-on camping experience. When they're older you, your parents and your children will all appreciate it more. I like another poster's idea about trying out the backyard first. You might get a mini-preview of how they'll behave. If you go, be sure to stress to the grandparents you want an agreed upon safety plan to help cut down on unexpected issues.