Is Your Backup Worthless?

May 13, 2014

I have been doing a lot of upgrades to customer Cisco Unified Communications Manager clusters recently.  Upgrades to 9.1 and 10 mostly, and a lot of people are transitioning to new hardware.  When we sit down and discuss the project, one of the first questions that I always ask is, “Do you have your cluster security password?”  In the last four upgrades that I have done the answer has been no 100% of the time.  Hold on, let me step back, three of them said, “I don’t know what that is” and one said no.  But they all ended up being a no because no one appeared to have it documented.

Now before you say, “well John, you’re the partner engineer, you should have this,” I will say that I agree.  Unfortunately, in all four of these cases the cluster was installed by another partner, I’m not going to mention who and play the blame game, I just wanted to explain why we are in the situation that we are in.

The last customer that I was working with on this started out thinking that this was probably just a minor imposition, after all there is a process (which I will cover later on in this post) to reset the cluster security password and it only requires rebooting the cluster to finalize.  They thought, so if we have to do this as part of moving to new hardware it’s not a big deal, right?  As they continued to try and remember the password so that we wouldn’t need to reboot the cluster the major issue that happens without having this password became quickly apparent.  My customer turned to me and said, “Wait, if we have an issue with this and have to rebuild the server then we can’t restore from backup, can we?”  Nope, hit the nail on the head there.  The truth is, if the server room burns down and destroys your cluster or there is some unrecoverable error on this disks or RAID array or some other unlikely but still possible scenario happens that requires you to rebuild and restore from backup and you don’t have that password then you’re screwed.  And it’s pretty sad that you’re screwed, because if you had one simple thing then you wouldn’t be in this situation.  And that thing is documentation.  Keep track of what that cluster password is.

Ok, I’m off of my soap box, let’s move on.  If you have read this post up to here and are now saying to yourself, “but, I don’t know what my cluster security password is, what do I do now?” then don’t fear there are a few easy solutions to this:

  1. Ask the person that originally installed it.  Hopefully they wrote up the documentation correctly and for some reason forgot to hand it over to you.  If so, then all is right with the world and you can make sure you have that documentation in a safe backed up location for that stormy day that hopefully never comes to your data center.
  2. Reset the password.  As I said before, this is actually a simple process to do, you just need to have a time that you can reboot your cluster after changing the password.  You do need to reset or your replication could get all messed up and we definitely don’t want that.  Here’s how to go about doing that:

Open a console session to your Cisco Unified Communications Manager.  You will need to do this on all the servers in your cluster, so just be prepared to repeat.  Keep in mind, I said console, not SSH.  SSH will not work for this process, so don’t even try to start there.  When you are prompted for login information, don’t enter your regular OS administration credentials, instead enter the following:

  • Username: pwrecovery
  • Password: pwreset

Once you log in, it’s going to ask you to remove any disks from the CD/DVD drive (if there are any in there), then put a disk in the drive.  If you are using a virtualized environment then you will need to mount an ISO to do this.  This is so that they can verify that you have physical control of the server.  Here’s a screen shot of the process:

pwrecovery

Once you have logged in, it’s a simple matter of pressing s to change the security password and entering the new password.  Repeat this process on all of the servers in the cluster and then reboot the cluster.

Simple, right?  So make sure that you have your cluster password documented so that you don’t find yourself in a bind somewhere down the line that you can’t recover from!

No Comments

Comments are closed.