Having not grown up with TV, and having gotten rid of the family tube when my daughter was in 5th grade, I can enthusiastically report that not having television is not a loss.
And so would my daughter, now mom to a 4yo boy who gets only occasional, carefully controlled video viewing. At my grandson's age, he would love to watch more, but we notice that his attitude deteriorates noticeably after sitting in front of the tube.
Consider the blessings: NOT watching TV means not being exposed to the constant barrage of commercial messages. Avoiding the constant temptations of consumerism is useful in nurturing the fine art of contentment.
And as you note, he would also have less time and attention for the expression of his own rich imagination, creativity, and originality. That could be a real and significant loss. Modern life is littered with distractions that can easily interfere with any of us focusing on our own gifts.
While it's true your son may be "deprived" of a familiarity with many cultural symbols, icons, and fads, he will surely become acquainted with them, simply because he will be growing up in this culture.
I've been without TV for nearly 30 years now, and my life is overflowing with worthwhile projects and interests. My husband and I have no idea how we'd fit more screen time into our satisfying lives. (And I also wonder whether your daughter is okay with you trying to introduce her child to television.)