It sounds like you and dad have put a lot of effort into helping your son improve at baseball --clinics, dad works with him, etc. -- and though you say you never push, and you surely are telling him "just do it for fun"-- he may be getting the unintended message that you and dad are pretty invested in his doing well at baseball, even if that truly is not what you meant to convey. Your son also is probably more aware than you realize that your area's kids' sports are highly competitive and I would wager that he hears friends talk along the lines of "When I make travel team" and "When I play high school ball" (not if -- when). Our area is the same way, and I hear parents talk about "Sally's going to do travel team soccer as soon as she's old enough" as if it's a foregone conclusion, etc.
So I hope there are other "pieces of the pie" where your son is involved as much as he is with baseball. Is he in Boy Scouts or another group where he can learn skills and do service? A church youth group, if that's your thing? Any clubs at school that are not sports-related, so he is well-rounded, such as a school Lego building club or chess club or science after-school program or drama or whatever? Or is the extracurricular focus just on baseball? By fifth grade in our schools here, there are many activities open to kids his age, and many are not sports-related.
He may be doing these things already and it just didn't come up in a sports question post here, and if that's the case -- great. But if his main or only thing is baseball, I'd consider continuing to support that but also ensuring that he sees you giving just as much effort to his Boy Scout troop, or helping with a school club he's in, etc., and that he knows you expect just as much effort from him in those activities as in baseball.
I talked with a mom yesterday of a soccer player. The kid wanted to try for travel team soccer because she's good but the mom pointed out: You also play in the school orchestra and like it (and is good at it); you are involved at scouting, etc. Travel team sports will end up taking priority over other activities period. The girl decided it wasn't for her; it's not a reflection on how well she plays but shows that she realized she has other things she enjoys too. If your son identifies himself solely as a baseball player, and it turns out he's a good player but not a travel team/high school team level player, does he have other activities he enjoys and does well at, so there's not a crash if he eventually is the kid who isn't on the travel team and clearly moving up and up in the sport?
He may end up a pro player; he just needs to be well-rounded and have other things going on. My daughter dances extensively (as many hours a week as most sports, and much more at certain times of year) and I am very involved with volunteering, helping out, taking her anywhere anytime, etc., but we also make sure that there is weight given to her other activities as well, so that she is not so focused on dance that if that went away -- and injury could take it, or a sport, away instantly -- she has other ways of seeing herself.