I have a couple pairs of very stretchy almost knit kind of fabric pants, one jean, one cargo style leggings. Almost any top / sweater and clogs or boots go with the jeans. I wear long, below my hips, tops with with the cargo leggings. I found those pants at a boutique, on sale and love them.
Honestly, I do wear a lot of yoga pants with cotton T's and nice thing swear shirts. I have an adorable coat from LuLu that is long, below my hips and it covers my bum in yoga pants. And I feel casual, but not so gym looking in that.
And check out stores that carry "Not Your Mother's Jeans", they are comfy, expensive, but you only need one pair.