My son went last summer for the first time (he's in 5th grade now). It was a five-night camp and he really had a great time. He's going back this year and my 7-year old is going with him. I think it's a really great experience for the kids - it teaches them a lot of independence, they get to make new friends, and they get to do a lot of outdoor activities that they probably wouldn't do at home.
When I was a kid, I went every summer for 1-2 weeks and looked forward to it every year. When my parents were kids, they each went for 8 weeks every summer and were just telling me last weekend how it was the best times of their lives. Both of them (at different camps) couldn't wait to go back each year - they saw the same friends year after year and loved all the activities, and the break from their parents.
Trust me - I get it with the time crunch. Baseball season for us will go literally until the day before my kids leave for camp. They do swim team all summer long and there are only 12 days between the end of swimming and the first day of school. If we're lucky enough to take a vacation at all, it will be for only 2-3 nights. But, I make sure to schedule a few camp-free weeks so we can do fun things during the day.
The camp my kids go to is Sunday - Friday, so it only takes up one weekend. Maybe you could find a similar 5-night camp for your first time. I really think summer camp is an amazing experience. As for family time, it will give you a week to connect with your husband.