My dad with me and my husband with my oldest boy did not completely remove the training wheels at first. Instead, they gradually raised them little by little until they were to high to touch the ground. When they noticed that the wheels did not touch at all for several days, they would talk the kid about removing them completely. "Hey, you rode all the way down the road and the training wheels did not touch the ground at all. Do you want to see if you can ride without?" We always let the kid decide when they wanted them off. Yes, my boy landed in the ditch a time or two and tore the soles off of a pair of shoes dragging his feet instead of using the breaks because he was afraid of falling.
What really helped my son with confidence was going to our church youth camp and being allowed to take his bike along. His bike still had the training wheels on but they were no longer touching the ground. The other boys made fun of his bike and he refused to ride his at camp. Instead he rode the other boys bikes throughout camp and when he came home he was ready for dad to remove the training wheels.