Most of my birthday parties have been combined with someone else (for the past 12 years, my SIL, as our b-days are a week apart). It would be nice to have my own party, that I don't have to plan!
I would suggest doing separate parties, especially if you're inviting friends. First, you want each child to feel like it's their special day. Second, activties that work for 3 year olds and activities that work for 6 year olds don't generally overlap, so if you're going to have games you'll want 2 sets anyway. And a 6 year old boy is going to want a theme like dinosaurs or transformers while a 3 year old girl will likely want Barbie or princesses. You could end up trying to run 2separate parties at once. Also, you don't want to many kids (a good rule of thumb is about as many kids as years in age, but a few more might be ok) or you're going to be overwhelmed. Just keep things fairly simple and inexpensive for both parties. (I spent $35 for my 8 year old's party, getting transformers masks, plates, cups, napkins, party favors and a pinata. Skip the pinata and you can do it for around $20. Make the cake, or better yet do cupcakes, and get inexpensive creamsicles instead of icecream. Less mess and cheaper)