I think, if you are new to a neighborhood, you want to tread very carefully. You take a risk if you approach a bunch of kids in defense of your child, because you can alienate the other kids and reduce the chances that your daughter will be accepted into the group. A nicer way might have been to really thank the host family (in earshot of the other kids), either when you left ("Thank you so very much for inviting us and helping us get to know some other neighbors.") or during the party itself ("Thank you for inviting us - Susie is having such a good time.").
If you are Asian and the children are Latino, you have many shared values and experiences, in that you have both been looked at as "the other" and had stereotypes applied to you by the white European majority. So you have common ground. You want to be very careful not to engage in anything that smacks of the treatment so many minorities have undergone - and unfortunately, many people just perpetuate the problem by inflicting the same dismissive treatment on others that they themselves have received. But since the kids spoke in English, I think it's likely that they did it just so you would understand them, to see what your reaction would be.
Maybe it would make sense for you to have your daughter learn Spanish (a good idea anyway, not just in this neighborhood). Anything that builds bridges is a good thing.
But she's 4 and they are 7 so they are not going to be in the same social circle anyway. So I don't think you have to confront every statement or get to the point of being so tired of a particular group that you feel the urge to intervene in kids' conversations.
Since you have at least some neighbors who are interested in socializing, then play to that strength. Next time, YOU have the party and invite at least some of these neighbors to that. See if there's any interest in organizing a neighborhood block party so that everyone can get to know each other. My neighborhood has one every Labor Day, with pot luck dishes and name tags and a food pantry drive. The kids do a lemonade stand & bake sale and raise money for a charity, and everyone runs around together with all ages having fun. Don't upset the apple cart by coming in and organizing things without cooperation, but do go to the mom who invited you and see what you can do to become more friendly and integrated with the families who have liver here for a long time.