Be firm. The thing is...you have to be willing to tell him NO, and then back it up. Honestly, it sounds like he's running the show.
After his regular bedtime routine, put him in bed, tell him to stay there and go to sleep, and leave. It's okay to tell him "Mommy and Daddy need some time to talk, and you need to sleep now." You know nothing's wrong with him and he's safe there. Even if he falls asleep on the floor, you can put him into bed once he's asleep...but he needs to do it on his own.
He's old enough to go to sleep without anyone there...and he's NOT scared, he's manipulating you. Get rid of the lava lamp, because it's just light to play by when he should be asleep (and is bad for the sleep cycle!). No night lights in his room, no noise, no light up toys. Put the light in the bathroom.
When I was a single mom, I turned the doorknob around so the lock was on the outside...so I could go get a shower and not worry about toddlers in the kitchen. But I ended up using it to keep my kids in their room at bedtime. As soon as they figured out that they couldn't escape, they went right to sleep. I unlocked it once they were asleep so they could potty if needed (strangely, my kids never got up in the middle of the night...they waited until about 5 am...almost as bad). Perhaps this is something you could do too so he's not standing at the door looking out. :-)
Best of luck.