Please go back and re-read Osohapi's great reply to you and then do just what is recommended there. It's right on the mark.
You and your husband seem to have communication issues if you cannot calmly discuss this rather silly situation together -- honestly you and he should be able to laugh together about these intrusive and sadly clueless folks but instead he's turning it on you and dragging in his larger argument that you're not social enough. That's a red flag -- are there other complaints he has about you that stay buried until things blow up as they did this time?
You do not have to socialize with people you don't want to cultivate as friends, just to BE more social. He needs to get that. You, too, need to get that you don't have to hem and haw and be vague if you don't want to do something. You and he really do need to talk calmly, and if he cannot BE calm and goes off again on making this all your fault (when the fault really is with the pushy sorta-neighbors), you and he need to talk about much more than one dumb potluck -- he's got resentments you need to find out about he needs to ramp back making this situation personal and all about YOUR faults in his mind.
As for these "guests," tell them you're not up for a potluck. If you just have to do something with them, suggest you all go out to dinner somewhere. Then you don't have them IN your house.
I would think that anyone who just walks in uninvited could possibly end up never taking a hint and would think they're your new BFFs if you do this potluck. Or they'll insist on reciprocating with a potluck at their house, wanting play dates for your kids (which you already know you don't really want) and more. Nip it in the bud unless you really want to get to know them. And meanwhile, YES, as Osohapi said, do start cultivating some real friends and reaching out to people you already know and actually might want to spend time with....Make that effort and ensure that hubby sees you're doing it, and don't put up with him bad-mouthing you as "not social."