As long as they're gaining weight properly, you can give them a bit of water. Our pediatrician pointed out to us that where we live, it gets super hot, and babies sweat just like everyone, but that breast milk or formula is mostly water. DD really only needed water on hot days (and then just a sip), or we would give her a bottle of water to hold on long car rides (my parents live 5 hours away - we didn't want a milk bottle to be left out unrefrigerated that long; we would stop once for a feeding, but if she was thirsty in the meantime, it would have to be water). Babies, by the way, are surprisingly good at regulating their nutritional needs. They know the difference between milk or formula and water. They'll only "fill up" on the good stuff.
Incidentally, that's how we got both of our kids off the bottle when the time came. They got milk or (diluted) juice in a cup, but bottles had only water. Made the bottles boring. But, they now both have sports bottles next to their beds with water, and one in the car (less likely to spill).