Could it be that your son is in need of a more consistent, empathic parenting style? Do you have consistent rules with definite consequences when they're broken? Emphasis on consistent. Every time he is too rough he is given a time out, sent to his room, or separated from his brother, the cat, the dog?
What did you do when he continued to color and looked defiant?
I suggest that you read Parenting with Love and Logic for ideas of how to provide consistent and realistic consequences for behavior, both good and not so good. 1-2-3 Magic is another good book to help with getting children's co-operation.
I do suggest that telling him to write letters instead of coloring could've been handled in a less controlling way. From your words and his reaction sounds like you challenged him which does usually result in rebellion. The two you were in a power struggle and he won because there was no consequence.
Perhaps you could've avoided this situation if you'd asked him to write his letters in different colors. Praise him by telling him you know he can write letters to make a pretty page, perhaps. Appeal to his ability to do well. Find out what motivates him.
The fighting on the playground sounds like he has anger issues. I suggest getting help from a counselor who specializes in children. If the school has a counselor that would be a good place to start.
Have you considered that he may have a developmental problem that is making it difficult for him to focus and pay attention? ADHD is just one such condition. He could have difficulty with sensory issues. Ask the doctor about this.
Being not challenged enough can result in not paying attention but it doesn't, in it's self cause anger and defiance. I've seen kids who talked too much, wandered the classroom, and were generally inattentive but they didn't play too rough or get into fights.
I suggest finding out why he's so angry and sympathetically deal with those feelings while providing clear and consistent rules and consequences. Not an easy thing to do. Taking a parenting class and/or reading some parenting books would help you find a way to deal with him.
I suggest that not allowing him TV or play is too extreme and is unrelated to the offense of fighting. It only makes him angry which detracts from his ability to learn a lesson. Try to find a consequence that fits the misbehavior. And your emotional involvement makes it less effective, too. I suggest that today was a bust.
First, try to back off, emotionally. Disciplining while upset is rarely effective.
Have you tried having quiet time with him talking about what happened and how to change it? I suggest some cuddle time with positive reinforcement will help him to allow softer feelings.
Didn't he have a consequence at school? No need for a second consequence but a huge need to find a way to relate with him so that he knows that you're concerned (not angry) and that you want him to learn how to get along with people.
Teach him how to get along by getting along with him. Teach him sympathy by being sympathetic with him. I'm not saying not to give him consequences. I am saying that it sounds like your focus has been on punishment and it would be more helpful to focus on teaching him ways to get along while showing him sympathy for the rotten way he feels.
Talk with him about what he can do,instead of fighting, when he gets angry. Tell him to go to his teacher or another adult on the playground and tell them he is angry and ask their help, for example. Suggest he run around the perimeter of the playground to get his angry energy out.
Role play with him, showing him how to get along. Have him help set up the role play. Ask him to describe a kid who makes him angry. Then have him be that kid and you show him how to get out of the situation without a fight.
It sounds like you have an adversarial relationship now. Find a way to get on his side so that he will work with you to make life better. It may help to have a counselor work with the both of you.
My daughter and her daughter were starting down that path with my granddaughter challenging everything her mother said, using a snotty tone of voice. They worked with a counselor and life is better. The counselor provided a way for both of them to air their grievances and deal with their feelings so that they could start over with some new skills.