I'm in almost exactly the same boat as you--my son is younger, but when I tried to wean him recently, or at least cut down the nursing, he just got clingier and worse, and it wasn't worth it to me to wean him. My oldest weaned abruptly, too, so I don't have personal experience.
It's obvious that he's getting comfort, rather than nourishment, so if you can find a way to exchange the act of nursing for something else, that might help. My older son steadfastly refused to be rocked to sleep unless I nursed him; and then when he weaned (in one day!) he just went into his crib rather than me rocking him to sleep, because I was trying to make the transition a little easier. His way was actually the easiest, but it kinda made me sad that he'd rather just lie down and fall asleep instead of having me rock him. :-( Anyway, if he gets attached to a blankie or some other "lovey" that may help ease the transition, because he'll associate the blankie with comfort, rather than just breastfeeding.
Since you're not ready to wean, don't worry about it. Maybe you can check out some "extended breastfeeding" yahoo groups or something. While you may get advice on weaning an older toddler, you'll also get a lot of moral support to continue nursing should you choose.
You may see if cutting out middle-of-the-night nursing helps. Let him cry a few minutes, and see if he will fall back to sleep like that. If you're worried about him being thirsty, put a spill-proof cup of water in bed with him. If you think he wakes up hungry, give him a bed-time snack so hopefully his tummy will be full enough all night.