I agree that he can be checked for color blindness but...please be aware that for many young kids, learning and repeating back colors is just not very meaningful. Sometimes a kid just doesn't see the point and is frankly bored by "What color is this apple?" especially if they hear it frequently, and they don't know that there's any point at all in listing colors. Same thing can happen at this age with letter recognition or number recognition -- and it's hard for a parent to tell whether the kid really has a learning issue or just does not care about naming those colors or shapes or letters.
I would stop doing much color quizzing with him, even if you're mostly pointing at things and saying, "What color is that truck?" He may be done with that and it's why he sometimes gets it wrong - he knows but doesn't care. I'd just stop doing it for a while.
Is he in a preschool yet? It's fine if he's not, right now, but I do think at least a year of a solid preschool before kindergarten is the way to go, for many reasons. One of those reasons is that he'll get more structured work on colors and numbers and letters, and also that he might care more about doing that when he's in a class of other kids who are all doing it -- when he's home alone with you there are so many other more interesting things around, but in a class setting he might be more motivated to do as others do and to please his teacher. Plus a teacher can tell faster than most parents whether a kid truly has an issue that needs investigation or whether a kid is just not that into colors....
I've known kids his age who were whizzes at one thing (could read some words, knew letters terrifically, were into trying to read, for instance) and who could not care less about another thing (like naming colors). Or who knew their numbers but didn't give a toss about letters. Or who just wanted to play and be more physical.....So don't let it worry you yet. I do suggest you look now into a good preschool for his year before he starts kindergarten, not just for learning, because preschool should be play-based, but also so he will get used to a classroom setting and get used to following directions from an adult who is not mom, dad or a caregiver. That is the single biggest thing preschool can give any kid before he or she starts K.