Well, I didn't have this problem my children were so happy to finally have room, they didn't like being swaddled. The nurses at the hospital found this very funny. Of course both my kids were 8 lbs, 15 oz, and 21 inches long. I am only 5"2' so they didn't have much room when they were inside. :-)
However, we did have a similar issue with the pacifier. What we did was let them have it at night, when we needed sleep; but during the day when they napped they didn't get it. It made them cranky for awhile and we had to get through that, but since we were all sleeping at night we were able to cope.
So my advice would be to swaddle at night, so everyone can sleep. Tired parents aren't good for anyone. However, when he naps losen it. Maybe get him to sleep swaddled and if you can losen it once he is out. See if that helps.
Lastly, you are approaching the 6 month -- get him to go to sleep on his own threshold. I really hated letting them cry it out. You may want to wait to deal with it all together. I will also tell you the crying it out worked with my daughter, few nights was all it took. It never worked with our son. He didn't sleep well until he was three. It turned out he had a mild case of sleep apnea which was discovered at 24 months when we were in the hospital for something else. At 28 months he had his adnoids out, and things slowly got better a few months later, and then finally by the time he was 3 we could leave the room and he would go to sleep on his own and sleep all night. Now at five you wouldn't know. Remember how well he sleeps at 4.5 months old, or 2 or 3 isn't going to matter when he goes off to college, gets married and has kids of his own. You not having enough sleep may matter A LOT when you fall asleep driving; stopped at a red light, or with something in the oven.
Good Luck!