S.W.
All the things the others have said may be true. Or, at 5 1/2, he may have been tired, hungry, upset because he didn't get candy or ice cream or a toy when he wanted it. He's just 5 1/2.
When he is none of the above, it might work to have a simple conversation with him about "when I pick you up from spending a weekend at "Dad's" I am interested in what you did and if you had a good time. You can tell me whatever you want, or not tell me. Only if there is something you are worried/scared/upset about do I really need to know, OK?"