I know this is going to sound harsh, but I'm going to say it anyway. HOLD YOUR CHILD! He is ONLY 6 months old. Babies cry for a reason and sometimes that reason is JUST because they want to be held and that is OK. As another responder said, babies need more then just food and a clean diaper to be healthy and grow.
Some children are content to sit for hours and hours by themselves. Others are not. It's not a bad thing. It's just who they are.
I would make sure there are no physical things going on. Look at your babies diet. Do you nurse or formula feed. Is he being given anything to eat in the afternoon that would upset his stomach and cause discomfort.
It could just very well be the fact that he MISSES you. You mentioned someone else watches him for 3-4 hours in the afternoon. He may be trying to tell you he misses you and wants to be near you. Do not ignore your child. Do not let him sit there and cry when he clearly needs you. A 6 month old is NOT trying to manipulate you. His ONLY means of communication is to cry, so that is what he does to let you know he needs something.
Yes. I would say he is having an attachment problem... he's not attached enough to you. As suggested, baby wearing is a great way to have your baby close and still be able to do the things around the house you want/need to do.
Please... listen to your child. He's trying to tell you something. He needs you.