I saw this after you added the So What Happened. He punched a kid in the face -- and this is on the "better" half-day schedule?
I would pull him out. Struggling through an entire school year this way -- he's fine one day, he likes school; another day he wants to stay home; another day he's punching a kid; if he's full-day he's frequently hitting kids -- this is just not wise. It will create so much stress that he will behave worse, not better. He will pick up on YOUR stress. I get that others are saying "It's too soon for the teacher to tell he should repeat K" or "He needs the consistency of staying in school" but I have seen parents feel they MUST stick with K when their child truly should have had another year to mature. He needs that. It is not a black mark against him for life if he stops now and restarts next year. It is not a black mark against you as a parent either! Go with your gut; it's clear your gut is telling you he's young and immature.
Do not just let him hang at home for the next year, though. I would look into preschools that allow kids to begin in January -- many around here have a January entry for kids who weren't ready (or in your case, were elsewhere) in the fall. You will need to do some serious explaining to the preschool about why he was in K and you pulled him out, frankly, but a really good preschool can work with him well. He does not need five full days a week of anything, either preschool or K; look at a three to four day a week schedule if possible.
He needs preschool (and lots of supervised play dates; and visits to library story times; and visits to library or rec center kids' craft classes; and rec center "wiggles and giggles" or physical activity types of classes for his age group) so that he can more gradually get used to:
Being in group settings.
Being told what to do by an adult who is NOT mom or dad, and obeying that adult.
Moving from one activity to another activity without melting down or expressing his frustration by lashing out at other kids or at teachers.
Expressing himself verbally and not physically.
These are all basic skills for K, and in the "old days" not that long ago, K was expected to teach kids all this. Today, K now is more like first grade was years ago, so K teachers do not have time to deal with socializing kids as much as they once did. Mostly that's OK, but with a child like yours who just needs time, time, time to mature -- it's not the K teacher's role to struggle through that, nor is it your role to push him to act more mature than he is developmentally capable of acting right now. Neither you nor the teacher can magically make him more mature; at his age that is a function of time combined with the right exposure to other kids and school-like or class settings. Please don't feel you "must" keep him in K just because he made a false start there. He is telling you in so many ways that he's not ready to share his mental or physical space with other kids in the daily setting of K. Listen to those signals.
By the way -- listening to his signals does not mean waiting for him to speak up and say "Yes, I want to stay/No, I don't" in any consistent way. You seem surprised that he "never gives you a straight answer" but he can't; he's too young and his mind truly believes one day that he loves school and just as truly believes the next day that he doesn't love it. Frankly, you can't trust a young child's statements and act on them every time because they change their minds and sincerely believe what they believe -- at that one moment, on that one day. Instead, follow the behavioral signals he is sending with his indecision and especially with his hitting. Pull him out, but give him a lot of fun, structured experiences in the next year. And have him continue with the counselor. If the counselor thinks your son would benefit from another year "at home" -- why would you not listen to the counselor? I'd pick the counselor's opinion over the teacher's at this point in time, especially after your son just punched a kid today.