Maybe someone could get better at your work and take over your job.
Maybe someone could take away your best friend or your sister away from you.
Maybe the colleagues who respect you so much would stop doing it.
It looks scary to lose what you have and where you are at.
How about this:
You will have to give up everything external one day. Why not build an irreplaceable internal resource?