I agree with Military Mom. Get a court ordered visitation agreement, and stick to it. Occasional shifts are fine for special occasions or circumstances, but the argument you use is that children need structure and dependability. But without a court order, you don't have to do what your ex wants.
If he doesn't want to see his child on his designated weekend, and if it's convenient for you to keep your child (meaning you don't have plans), then you can keep your child. But the next time your ex wants to see his child, it's on his next scheduled weekend. If he contacts you in plenty of time and says he has a business trip or his mother is ill, you can agree to a swap if you want to. But the reverse is also true. So go ahead and say that, no, you have plans to be away for a weekend (even if you don't) and you can't switch and he needs to take his child as scheduled.
If he's trying to control your life and schedule by constantly upending the plans, then no, don't go along with it. And if he's displaying a ring, ignore it. He's baiting you. If he had respect for you, he would tell you. But he doesn't. So don't take the bait by commenting on it. It's not your business, and your social life is not his either. You don't have to explain why you cannot switch - just say you have plans. No details.
Sure, your child wants to see his dad - just say, "This is our special weekend together, and next weekend is your special time with Dad." Make a calendar your child can view that shows the schedule.