Whether or not you stay is decided by the host's preference and your comfort level. I like Momma L.'s advice to stay awhile and see how it goes.
When you RSVP I'd tell the host about your son's food allergies and ask if you can bring an alternative for him if she's serving foods to which he's allergic.
Is he so allergic that he'll need the epi-pen. My granddaughter has allergies, including to peanuts. Her mom doesn't send the epi-pen to short social occasions such as birthday parties because it's highly unlikely that she'll need it. She's 11, has had a pen since she was a toddler and has never needed it.
If you feel he needs to have the epi-pen with him then you need to stay. It's too much to expect someone untrained to be comfortable with having it and using it.
It's best to be up front with the host about his allergies. And.....to teach your son what he cannot eat so that he can help monitor them. My granddaughter from Kindergarten on would ask if peanuts were in foods before she would eat them. She knew she couldn't have milk. Tho that allergy is not life threatening, she would refuse it.