Children absolutely need time to be alone, to observe their surroundings in a natural state (rather than through activities), and to just BE. They need down time, they need time to think and to develop a good sense of self. There is such emphasis today on keeping kids engaged all the time. Some parents are even trying to potty train 1 year olds and get 2 year olds to write their names. Then, once they are in school, the kids are shuttled from one organized activity to another after school and on weekends. They cannot entertain themselves and are not at ease in their own skin. You already see that your child needs to spend time in the swing alone and quiet. You can face the swing towards the window so he can see the trees and maybe the birds - you could put up a feeder even, but at least it's not a structured activity. He can be in a play gym or something where HE is controlling what he does and what he looks at. He needs some things that are NOT at a frenetic pace. The early childhood and behavior experts will tell you that the children who have the toughest time in school are those who are unable to work independently, who cannot focus on an activity without adult supervision, and who need to be the center of attention. If your child is already showing signs of irritability, it could be connected to overstimulation. He needs time to process his day and his activities. Don't be pressured to do too much - allow him to slow down.
Our son was only allowed to have 2 structured activities in his early grades - he had religious school and one other activity of his choosing. He experimented with soccer, basketball, etc. - he got to try a bunch of different things at his own pace. The rest of the time, he played with kids in the neighborhood or occasionally a planned play date with someone from across town. Sometimes we went to a movie or a museum, but other times we just explored the neighborhood casually. He spent time hunting for toads in the yard, experimenting with building things from various toys he found around the house (mixing up things from different sets), and had time with his thoughts. He developed the ability to be creative and thoughtful. He has just entered an environmental engineering program which is a perfect mix of creative thought and structured discipline. We couldn't be more proud, and he couldn't be happier. He absolutely was not harmed by having fewer activities than the other kids, and he did better in school than most of the kids who had something different every day. Go with your gut!!!