You should generally not have a password access from anywhere, instead load the key physically onto the machine - this could be via USB, a CD (which has the bonus of you know no one has meddled with it after you've written it) or if you're really hardcore - hand type the whole key in.
If you use the 'password access from another server' route, how do you make sure you've secured that server enough?
This is also known as 'out of band' communication. You don't share authentication secrets over the same communication as you'll be using them on. As you can never be sure prior to authenticating that they are secure.