This will really depend on your daycare provider. If they are willing to take him, I'm not sure that he shouldn't go. I have a feeling that you could spend another week home with him and have the exact same issue the following week. As far as being worried, try not to make yourself sick! This is something he will battle for awhile and work through. I don't see any reason to go back to the doctor, he will just tell you the same thing. When my son started daycare, I was at the doctor constantly because he always had a fever! My doctor would never do or say anything differently: make sure he drinks a lot and give him motrin. My son also did not seem affected by this persistant fever at all. If it spiked before I gave him his next dose, he would get cranky. Once medicated again, I couldn't keep up with him.
As my doctor was examining him, he was smiling and laughing. The doctor then told me "if he is acting like this, he is not truely sick!" He said that when he is miserable, not drinking and not eating, that is when more attention needs to be given.
I know how hard it is to see the love of your life sick so often, but it will pass. If daycare will take him and it isn't a high fever, I say go for it. If you trust the provider, you could also provide them a measured dose of tylenol to help him through the day.