This happened to me almost 15 years ago and I found an easy solution.
I shave with the grain, not against.
In other words, your hair lays down a certain direction. You must shave going in the direction it lays down. If I'd shave the opposite direction to get a closer shave, I'd suddenly have a ton of bumps everywhere and it would be sore and irritated.
It never happened until after my baby was born - and it seems odd, but my recommendation is to let the current bumps calm down (don't shave for a while until they aren't as vivid and irritated), then try shaving the same direction your hair is laying.
Also, I only use non-scented soap. I soap really well and am sure to shave very "wet" so I don't get dry rash. I also make sure to follow up every shaving with a non-perfumed lotion.
I now have beautiful skin where I shave!
Again - the secret is in making sure your skin is back to normal before you start this technique or the rash will continue.
Good luck to you!
S.