P.G.
Don't push him. You both need to be on the same page for this one - there are too many lives involved. Don't talk yourself out of it though. I had my little guy when I was 39 - you do have time. I know several families who have a 5-6-7 year age spread between kids and they LOVE each other bunches. Enjoy your daughter and husband, let him and yourself off the hook about it.
I know there are articles out there about this issue - read up on how others deal with it; you may find some ways to think about the situation that help you feel less sad.
Also, your little girl is about to start school, which is going to open up a WHOLE new level of busy for you, and that may be a place where you can put some of that love and energy as a room mom or something.
Good luck!