The big deal with organics is that you get less preservatives, and often you get less sugar and other "yucks".
Also, some people worry about the chemicals that farmers put on their crops getting into the food supply. For instance, a farmer sprays Round-Up on his strawberry crop repeatedly to keep the weeds from growing up around them, and then the strawberrries are picked and put in the store. Some kid comes along, picks up the strawberry, and eats it without a thorough washing. Yuck. Supposedly, it's safe, but many people worry that there will be adverse effects in the long run.
With organics, this is not a worry, as they are grown without pesticides and unnatural fertilizers.
Certified Organic foods are watched FAR more heavily than any other food. They must undergo rigorous certification and are checked up on reguarly. Conventionally grown crops, on the other hand, are almost never checked...only a very small percentage of the food in our conventional food supply gets government inspection. Mostly, the government leaves companies to check themselves.
Babies are tender little things, and so many people buy organic baby food even if they don't worry about it for their older children.