I too have both grasses, one in the front, the other in the back which we did on purpose because of sun/shade ratio. I wish I had never planted the St. Augustine because of the problems I've had with it and the extra time/cost to correctly maintain.
You can research it on the web but what I've learned is that it needs more water due to the shallow roots, should be mowed more frequently and is more susceptible to disease.
Two years ago we got big patches of dead grass in the St. Augustine that we tried to get to grow back by fertilizing and pulling the weeds that started to grow there and we still have bald spots. We have now hired a company to help bring it back. The only place it really looks good is the east (less sun) side of our house.
The Bermuda is one tough grass that as long as we fertilize on a schedule, mow once to twice a week and water deeply about twice a week we never have a problem with it. As far as the flower beds go just spray round-up by the edge of your beds (careful around other plants) and that will keep the grass in check. This grass won't grow in very much shade though.
Any grass, including shade types, is going to need at least 4 hours of sun. For heavy shade try a ground cover instead.
Maybe you should try the Zoyzia!