Been there, done that. I NEVER use the stuff people give away or lend. I just keep it in plastic storage bins in the basement, attic, or garage. When they want it back, it's ready to go...presto!
The last thing I want to deal with is hearing about the bad condition the clothes are in when they come back, or like you said, worrying about trying to track the stuff down.
Lastly, if you refuse a gift of used clothes, it can get pretty ugly...as these people for some reason take offense and think you're trying to say something about the cleanliness of their home, their taste in clothes, or that it ruins their chances of being "generous." So on that note, don't say anything. Just take it with a big smile and a happy "Thank-you" and store it where ever you can and where it won't get damaged. In time, you can ask if they want em back because you're going to donate them to the Good Will. They might freak out on you, even then, but at least they can let you know if they want them back before you get them out of your hair.