You might want to look at preschools. My son was in Transitional Kinder and his old preschool had a program that picked him and a few kids up and they played at the preschool (the were not with preschool kids, rather in their own group). The preschool is play based, but I am sure there are more academic schools around too. The public school did have an after care program, yet I felt there were too many kids to teacher ratio for his age.
edit: Could you have someone go to the school and teacher your son some enrichment class. It could be piano, guitar, chess. Often you could get a group of parents willing to hire someone. Two times a week with a Spanish tutor and 5 kids. This will allow your son to stay on campus (no needless driving), he could be with some friends and learn about a subject. My son's school has after school enrichment for a small price. Gardening, chess, drumming are just some of the options.