I used to do the same thing. My son would not go to sleep unless I was rocking him, dancing him to sleep, or if he was in his swing. As long as he is not in it when no one is around, I think it is ok.
As my son grew too big for the swing, I got him accustomed to a different way to fall asleep which consisted of me or dad reading him a book or singing to him softly while rocking with him on the rocking chair, and then as he nodded off, put him in his crib or bed. As he grew older, we just read him a book or sang him some soft songs and put him to bed. If he fussed after we put him in bed, we would rub his back or tummy and either softly talk to him, or sing him a song, sometimes we even put on some classical music and that helped him go to sleep also. It really depends on your baby's personality. Some take transitions easier than others.
Maybe you can try rocking him every once in a while or singing, reading a book, or playing classical music. Even if you do this gradually such as reading, playing classical music, or singing to him while he falls asleep in the swing. This might make the transition easier. Sure, you might end up having to sing, read a book, or play music every time before he goes to sleep, but there is nothing wrong with that. ^.^
If your son sleeps in a swing, let him, but try different methods that I had suggested (and maybe what other mothers suggested as well) so that there is not just one way he will fall asleep.
Another thing too is that when reading you can involve your three year old. Even with singing, you can ask if he can help sing with you so brother can take a nap. Even if he does not know the words and just hums along, that would be wonderful. Or if you can trust him to rock with his brother in the rocking chair with your supervision of course. Also, if you play classical music during nap time, tell your three year old that when the music is on, it is quiet time, or resting time, or nap time. Just try a few things and find out what works best.
M. *~