Please put the fury on ice if you can.
He goes to day care (even if it's only once a week); he has sitters at the YMCA; and there are other kids in both locations. It's just as likely that he heard "brat" from another kid as from an adult - will you be equally furious if you knew that he heard it from a kid? I think you're assuming here that an adult directly told your son HE was a brat. That might not be the case at all and you really have no way to know.
Also, you mention "the time he told me he was bad because he had an accident" as well as this time when you seem upset that he's blaming himself or criticizing himself. This is pretty typical for children his age, so don't let it either make you assume he's been treated badly by an adult, or assume that he is unhappy or too self-critical. Remember -- for kids his age, the entire universe still revolves around themselves; that means it's "all about them" and that can include the idea that "If something went wrong it was my fault" and "If someone told me I did something I shouldn't have, I must have been very bad and naughty." So...even a "no, we don't do X" from an adult, said nicely, can turn into "I'm such a brat" or "I'm so wrong" or whatever. Again -- this is how kids this age see the world, and it is typical. He will outgrow it, if that's what worries you.
But the brat thing -- let it go. If he continues to say it about himself over and over, yeah, ask him gently where he learned that word and why he says it about himself when he's a nice and smart boy. But otherwise, let it go and save yourself the stress of being furious.
One last thing -- does he watch ANY kids' TV at all? Brat is not considered at curse word by any stretch of the imagination, and I feel sure it likely turns up in many a kids' show, so don't overlook that possibility.