M.P.
I think the entire home environment is a negative influence and definitely a part of the reason for his talking back. I strongly recommend that you get him involved in counseling. I suggest that you go back to the court and ask for a change of custody and a new custody study.
You did not mention how you discipline around this issue or if you have any other behavior issues going on. You can positively influence your son while he is with you by using an effective discipline style. I highly recommend getting the book Love and Logic by Foster Cline and ------.
Focus on all that he does that is positive by giving lots of praise. A parent needs to give 8 positive remarks for each negative one to be effective in building self-esteem.
Talk with your son about what sort of language and attitude is acceptable. Allow him to vent about anything that he wants. Make it a discussion, asking him what he thinks. Keep a positive and calm demeanor while remaining firm at the end that you will not accept him talking back to you. Decide in advance what the consequence will be for doing so and be consistent in applying it.
If it were me and my grandchildren, I would tell them to go to their room until they can speak in a respectful manner. They have the option of coming out and starting over when they know that they will be respectful.