Is he drinking water? Sounds like maybe he's just dehydrated, and since he's not getting a lot of fruit/veggies with high water content either, that might be the issue. Try giving him a sippy cup with water in between meals....
It does sound like he could use some more fruit and veggies, based on his typical diet (he ought to be getting at least one or the other with every meal AND snacks) - fruit is good to mix into baby cereal or for them to eat on their own (blueberries, strawberries, bananas, raspberries, prunes, etc). You can mush it up or just give him small pieces to feed himself. My son loved prunes! Peas and corn are good - there's no reason not to try green beans, avocado, broccoli, carrots (cooked), zucchini/squash, winter squash, sweet potatoes - anything with a dark color (green or orange) is great for him. Most of those suggestions are also available bagged and frozen (i.e. - cheap!). This is the time to get him used to those kinds of tastes so that he'll continue to eat them as he grows....
P.s. - he really does not need juice at this point, unless you absolutely can't get him to eat any fruit or veggies. It's typically just a lot of unnecessary sugar and will fill him up rather than leaving room for the healthy stuff. If he's REALLY constipated, go for prune juice (don't assume he won't like it if you don't - kids this age are typically really open to most foods!)