This is a birthday party for 5 and 6 year olds which means that you don't invite a 3 year old. What is appropriate (ie. fun, safe, and skill level) is not necessarily appropriate for a 3 yo.
I also add that it will be difficult enough managing 5 and 6 year olds and having a 3 yo, even if he's well behaved adds to the difficulty.
My daughter has a birthday party for her kids that has only kids near in age. And then the family, which includes younger and older kids have another one which is not as structured as the same age party. This year the family party included my daughter's friends and their kids. She rented the gymn at a community Center which included an adult to manage games. Each child could choose what they wanted to do. There was basketball, trampoline, a giant ball to push around, "bowling" played on the floor. And of course just running around which is what the younger kids mostly did. The adults set up pizza, cake and ice cream in the kitchen while the kids played in the gym. There was usually 1 or 2 adults in the gym with the adults rotating in an informal way.
My final word is that you do not need to feel bad about not inviting the 3 yo. As children grow older they have different skills, likes and dislikes. The 3 yo does not have the same level of skills and does require more supervision even if they are well behaved. You already know his mother isn't able to provide that supervision.
I also agree that you don't have to tell this friend about the party. However, if the 2 families see each other often it may come out. In that case it's best to mention it ahead of time and tell her the party is only for 5 and 6 year olds. She shouldn't be upset.
If you want the two families to celebrate your son's birthday you could invite them over for cake and ice cream on a different date.