We had this problem mostly with our first daughter when she was about 3 1/2. I recall that when I was a child I also got up in the middle of the night--in response to nightmares--and went into my father's bed or my grandparents' bed.
Given my own history that it was in response to nightmares, my husband and I tried to be compassionate about the situation, but you do have to set limits that allow you to get rest. If your son crawling in bed with you isn't waking you up, to be honest, I wouldn't view it as a problem. He will grow out of it. I don't crawl into my grandparents' bed anymore, haven't in over 3 decades. ;-)
In the case of my daughter, it was becoming an issue for my hubby and I because at the time she was doing this, our 3rd child was a baby, and he slept in a cosleeper next to our bed. When he would need to nurse, I would pick him up and nurse laying in bed, often falling asleep with him there beside me. When we added her into the mix, the bed was too crowded. But like with you, my husband and I often didn't wake when she came in.
Depending on your son's age, this may or may not work. We set up an incentive system for her to stay in her bed. I put a calendar on her wall, and bought a small accessory set for her dollhouse which I hung on the wall beside the calendar. I explained to her that each night that she stayed in bed until morning, I would mark the calendar. When she got 10 days marked in a row, she could have the dollhouse accessories. My husband and I did not get upset with her if she didn't make it through the night in her own bed. But the calendar did not get marked, and she was back to "Day 1." It took just a bit longer than 2 weeks for her to earn the dollhouse accessories.
We tried this first not having the toy already purchased--telling her we would take her to the store to get it--and that did not work. I think it was too abstract for her. She needed to have that toy where she could see it and long for it.
I've heard of other parents making a small bed on the floor and telling kids they could sleep on that rather than in the parents' bed. I know that my grandparents eventually told me that I should sleep on the couch right outside their bedroom door (their bedroom was off the livingroom) if I woke up at night. I was having nightmares as I said, and just being that close was comforting to me.
Oh, I should mention that I have 4 children--2 girls, 2 boys. The girls sleep together even now at ages 8 1/2 and nearly 7, and the boys are 5 & 3 and sleep together. Not sure the ages of your two, but you may find that they sleep better together. I wouldn't worry about mixing genders if they are young--my first son slept with my daughters until the second son was about 1.5 and then the boys started sleeping together.