I might be in the minority here, but I think you should give it a little more time for the adjustment phase. Pulling him now will likely only enforce the idea that school is something to be afraid of and also give him the impression that crying and making you feel guilty will earn him a reward in the end (getting his way). I'm not saying he should have to spend every day feeling sad and alone, but he's still going to need to learn to cope with life without mom around all the time.
With my first born, he was in daycare from infantcy (I worked outside the home). I think he thrived immensely from time away from me. I'm now a SAHM and have a 7 month old. I've already been contemplating working part time, if for no other reason than to be able to afford daycare a few times a week. I would likely not be working for any money, because it would all go to pay for daycare. But I'm so worried of this very thing.....the clingy child who can't function in social settings without mom.
I do agree with some of the previous posters....try spending time in the class with him and then gradually decrease that time until he's a little more comfortable. But overall, he needs to see YOU comfortable in this situation as well. Your anxiety will feed his anxiety.
Good luck and try to hang in there. Its one of the hardest things to do, but I hope I gave a little insight without coming off like there's only one right way to do it.