Oh L.,
I know you love the job you have now and it is a great spot for you, your experience and your personality.
What are the major differences with people you are accustomed to being with daily? Is it more behind the scenes with making things happen where you are not as involved with some of the people you love being around?
I believe I have a hunch as to why you might be holding back and quite honestly, that would be a reason that I would have a hard time with as well.
As far as the company, it is very stable and if you take this new position, you would likely be there until you choose to retire.
My thoughts are with you. Like you, I would be torn.