Padding serves any of 3 purposes: 1-impact protection, 2-snug fit, and 3-comfort.
1) If it is removable, then it is not required to meet the impact protection legal standards and is fine without.
2) For small babies, the padding is necessary to get the baby fitted snuggly in. If he can be fitted snug without it, then you don't need it for this purpose either (but you can keep it in if you like).
3) Some people leave the padding in just to give the baby extra soft comfort. Some of the head pillows I've seen look uncomfortable (push the head too far fwd) and are not necessary.
I kept my babies in the infant carrier until they hit the weight limit b/c it was SO convenient for me with the base in the car (they never reached the height limit). But if your son looks big, look at the distance between the top of his head and the top of the carseat--if there is at least 1" of seat above his head, then he still fits. Legs do not have to be "inside" the seat. If you choose to move him to a convertible seat, the same guidelines for the removable padding apply.