Allergies can definitely affect hearing, especially at such a young age. Even though your son's "hearing" can be tested and found to be fine, this doesn't mean that he's hearing clearly. My son also had delayed speech, but had several ear infections, and we decided to have the ENT put in tubes. We had been told that he hears sounds like he's underwater as a result of the fluid. THAT was a wake-up call.
Kids that have fluid in their ears, whether from a cold, infection or allergies, do not hear as clearly as most of us do. It can definitely affect speech. Normal "hearing" tests won't pick up on clarity; they usually test beeps, not words or speech. Both of my children have good hearing, but have central auditory processing disorder. One of the red flags for this is frequent ear infections when a child is young.
Unfortunately, each child deals with this issue in different ways. My daughter would argue with me that I said something I didn't, whereas my son just started tuning out most of the sounds he deemed unimportant. It's exhausting to put in a lot of effort to hearing and understanding what is said when it's garbled.
Even after we discovered the auditory processing disorder, my kids still needed speech therapy, but they are much better now. If this is true for your son, school will present many challenges, and you (and the school)need to know about them before your son faces them and finds his own way of dealing with them. There's a book that helped me called, When the Brain Can't Hear. It's worth reading. Good luck.