What is his height percentile? Is he growing, even slowly?
My niece was so small! She didn't hit 20 lbs until she was a little over a year, so she had to ride in her infant car seat that whole time. She wasn't a good eater, but mostly, she was just small. Then she got a stomach bug. Because she was so tiny, she ended up in the hospital. After that, they tried everything to get weight on her, including finding the fattiest, highest-calorie butter spread in the store, and putting it on and in everything. She did put on some weight.
I think the worst after-effect of this incident is how she and her little brother eat now. They're super picky still, and since my sister and BIL were desperate to get them to eat, it because a huge battle, with the kids manipulating their parents like crazy. When they didn't want to go to bed, they suddenly said, "I'm hungry" and their parents instantly stopped putting them to bed and let them stay up late to eat. This turned into their not eating any dinner, which they didn't like so much, and waiting to eat until after bedtime, when their parents gave them yummy cereal instead of that yucky broccoli from dinner.
My niece is still small. She was wearing size 2 shoes at age 10, and my 6 yr old is wearing size 1 1/2 now. My sister worried about her constantly, but did some family research and found that we have a tiny spitfire of a woman in our family tree. She was only 4' 9". that helped my sister stop worrying. Also, my nephew, although he ate just as poorly, is plenty tall.
Stick it out, giving him lots of things to try to eat. Cheerios, those fruit-puffs (they're fun shapes), and puree things with a hand-blender to make them extra smooth. But don't let him know that you'll do anythying to get him to eat, or you'll just make more problems for you and him in the future. It sounds like he might have sensory issues, where he doesn't like textures. Does he pull at his clothes or hate tags in his shirts?
These are my random thoughts on the subject. Sorry they're not real organized!