I have a cat that also lives outside. She was an inside cat but she decided to use my entire house for a litter box no matter where we put her litter box or how clean it was. She prefered my carpet. I love my cat, but I can't afford to have her going all over my house! Now she is mostly a garage cat, but she can come and go as she pleases. It was either that, or we would have had to put her to sleep, and I can't do that. If it bothers you that the cat comes in your yard, then there are things you can spray around your yard to keep her out.
I personally am more peeved with the people that let their dogs pee on my mailbox plants when they are on their walk, or let them out to poop in my yard so that my kids step in it. My cat at least will now use her litter box.