It's not unusual for a shy child to be selective about whom he talks to... and also whom he shows off his accomplishments to.
It never hurts to check with your doctor; if there's no medical problem, there's one less thing for you to worry about. But three-year-olds often display this behavior. Shyness is sometimes a temporary thing - just a little uncertainty around strangers (including family) - and other times it's a built-in personality trait. A naturally shy child needs to be taught in a different way how to interact with people, where a more outgoing personality sometimes needs to be reined in a little!
There are little children who do well at home and with a small group of peers, but who simply shut down in a large group. If you're mostly used to kids that jump at the chance to be with a lot of other children, this behavior can be baffling.
Talk to the teachers some more, and let them know what he does at home. If they say they understand and show that they can deal well with his personality, give them a chance to do it. If they insist that he's not learning and not bright, you might want to look for another school or daycare arrangement.