It depends what they can get into. Do you worry about them underfoot when you have boiling water on the stove? Are there medications and cleaning chemicals accessible? Do they have smaller friends who come to play? Is there a huge tv not anchored? If so, you may want to keep the gate to contain them at certain times.
But in general age 3-4 is proabably time to get rid of it. Our's are 3 and 7, and we have no more gates downstairs. We have one at the top of the stairs we only close at night to prevent a fall in the dark. We also have a key locked door to our unfihished basement.