I went my entire life eating nuts, PB&J, Reese's cups. Several years back I ate a PB&J and had to be rushed to the hospital. My face was swelling, I could hardly breathe...it was terrifying...I was going into shock. Thank God, we live a block away from the hospital. The thing is, I HAD been tested for nut allergies, and it came back negative. I can attest that what the first Dr. told you can absolutely happen. The skin test could be right or wrong, but you absolutely need to know. It's true he can be fine one day, and having a life-threatening reaction the next.
Get the blood test. I will give you an answer for sure, and you won't have to go through what I did.