This question gets asked a lot on Mamapedia, and the prevailing wisdom seems to be that a) goody bags are getting out of hand and b) $20 per guest is ridiculous even with only 6 kids. Can you imagine the huge parties with 25 kids???
Don't fall to the pressure of the mom who goes overboard. Maybe she did more because she had 2 kids and they were each receiving gifts, but still, $10 per party favor per child is still too much in most people's estimation. It just aggravates all the other parents who are trying to keep this under control. I commend you for doing a small party with just a few kids - that's extremely appropriate for a 4 year old!
A party favor is supposed to be a small memento of the event, not a quid pro quo for the guest bringing a present to the birthday child. Somewhere along the line, this whole thing got way out of hand, and a lot of moms are feeling like you do! Kids wind up wanting to attend a party to see what they're going to get, instead of going to celebrate a friend's special day.
I don't think it matters that you meet up with her for play dates - you don't have to "pay" her kids for that just because you see her frequently. Resist the pressure, start a trend in your social circle (or more accurately, return to a sensible custom from years and years of party-going and party-giving.) Some people just do a simple craft, and the finished product is the "favor" - kids do pictures, or decorate a picture frame, or create something else. Or they just play a few games.
I think, if you keep it simple, you'll set a standard for all your parties to come.