I wore my son a lot the first year, less the second, and yes, hard on the back.
He had no trouble with spine or hip issues, but was also seeing pediatric chiropractor every few weeks since he was 2 weeks old, so that may have been preventative in that regard.
He was 'later' in things like sitting up, crawling (around 9 months) walking (15 months) and such, but I am not sure that it is wise to be in such a hurry regarding that sort of development. I should say that he was on the 'later' end of the average spectrum for these activities, I didn't worry about it. He always felt very cozy and safe and liked being 'up'.
I cannot say that confidence issues stem from babywearing-- I think that has a lot more to do with temperament and feeling secure within an environment.... I nannied for many families, some of whom wore their babies a lot, some which didn't, and saw the gamut of temperaments emerge as the children grew older.
So as long as you are feeling 'balanced' in how you wear your son, if you are ensuring he has time to "just be" in his moses basket, or on the floor... that he's not CONSTANTLY on you... I think that's what's important. My son was either in the pack or on floor pads (layers of firm quilts, not puffy) or in his basket and he did just fine.