Part of attending a private school (having grown-up in one) is understanding that in the end the Board of Directors/Trustees is the governing body. They do not have to justify anything to the parents, especially if it would involve releasing information regarding Human Resources matters.
You don't say what actually happened, so either you don't really know (gossip isn't accurate) or it's so aggregious that you can't share it, so it's hard to judge whether or not you should leave the school.
MOST private schools have a "morality clause" in their contract, meaning that if the Board decides that a person's actions are out-of-line with the vision/mission statement of the school, that individual can be fired without warning and without the need for justification. Sounds like this is what happened.
Each school has different bylaws regarding the Board. At this point, the best you can do is appoint different people when the terms are over and as a parent body, demand a review of the bylaws.
For what it's worth... the prep school (100 years old and nationally-ranked) I attended was nearly ruined in the span of 6 months by an incompetent Board and the criminal they appointed as the President of the school. There was NOTHING that the parents could do about it, but they put pressure on the Board to actually do a background check on the guy. In the process... found out that he had driven his previous school into the ground and was sleeping with one of the Board members on the interview committee. On top of that, they had issued him a cell phone and vehicle with GPS (pretty standard for a prep school president)... checked the records which revealed that he was giving out his cell number to female students and taking them on unchaperoned trips. That "morality clause" sure came in handy!
One way that we (alum) made our voices "heard" was to withold money. The president of our Alumni Association sent out a mass email encouraging us to hold our annual donations until the situation was resolved. We're talking millions here... most of us give several thousand a year to different aspects of the school... 100 years worth of alum holding out their donations. The Board moved pretty darn quickly.
As a parent body the only real money you can hold is your tuition, which may come back to bite you. See if you can get the word out to alum and encourage them to use their financial "voice"- it often speaks much louder than a rally!