danejeffrey.com » snapshots

My Favourite Topic (or Yet Another Lesson in Snapshot-ology)

danejeff — Thu, 29 Apr 2010 07:18:17 +0000

Another snapshot post. Who new snapshots could be this awesome.

Again, the situation is VCB failing to remove the snapshot it took of a VM when the backup job that kicked it off failed (because of lack of disk space in this case).

First, vcbSnapshot was used to try and remove it via the VCB proxy manually. For one reason or another, this failed and a Consolidate Helper- 0 snapshot is left behind. In the VI Client, it will show no snapshots in Snapshot Manager but a check of the VMDK targets show that the VM is indeed pointing to delta disks.

Tried taking a new snapshot manually to confirm the .vmsd file is not corrupt. The chain appeared good. Used:
vmare-cmd .vmx removesnapshots
and received the error:
unable to access file since it is locked

While looking into the problem, found these excellent resources for the undocumented vmkfstools -D switch which dumps file info to /var/log/vmkernel. It can be used to identify the owner of a file lock which could be causing the above error when trying to commit back the snapshot delta files:

http://vi3.org/download/guides/english/vi3/StoppingaVMandReleasingFiles.pdf
http://blog.core-it.com.au/?p=477

In this case, however, I didn’t go through the process – instead took the coward’s way out and used: service mgmt-vmware restart on the host that the VM was running on and then used vmware-cmd to commit the snapshots – worked fine this time.

So, I would expect that if I had used the -D switch on the VMDK’s attached to the VM, the owner would have eventually been the ESX Host Agent which would be locking the file on behalf of the VCB request. Must remember to go through the process next time to find out for sure…

VMware ESX – Another Snapshot Fix

danejeff — Sun, 25 Oct 2009 04:23:28 +0000

As an update to this post, a snapshot chain could also become corrupt if for some reason the .vmsd file (snapshot definition file) does not get updated when a snapshot is taken. The result of this can be seen by running:
vmware-cmd .vmx hasnsapshot
and it returning blank, or by checking Snapshot Manager in Virtual Center and it being blank, however you know snapshots exist because when you check the VM’s .vmx file, it is referencing snapshot delta files not the parent vmdk.

The .vmsd file’s purpose is to maintain information about the snapshots on that VM. It is read by Virtual Center to get the snapshot name and chain details. It is also read by vmware-cmd in the above command.

The solution is is to first confirm that hte current snapshot CID / pCID chain and fix that if it is broken. Then remove the .vmsd file. Next, create a snapshot on the VM either through vmware-cmd or through Virtual Center (it’s best to do this logged into the ESX host directly rather than Virtual Center because we will also remove it from the host). Next, check the snapshot chain either in Virtual Center or using vmware-cmd, this time you should see hassnapshot = 1 and the fiull, correct chain in Virtual Center. Finally, if the chain is restored correctly, remove the snapshot chain as you would normally.

VMware ESX – Fixing Broken / Corrupt Snapshot Chains

danejeff — Thu, 17 Sep 2009 12:12:40 +0000

Have recently had to deal with a virtual machine regularly failing to remove snapshots via VCB. The VM has 3 VMDK’s connected to it and I’ve seen as many as 11 delta files per disk (through VCB and troubleshooting efforts, etc) – that’s 33 snapshot deltas in the working directory!

Anyway, what can happen is the snapshot chain gets corrupted somehow and needs to be consistent before ESX can remove commit the snapshots back into the parent VMDK. You can see this happen when you run vmware-cmd .vmx removesnapshots and it returns quite quickly with an error.

What to look at:
You need to go back through the VMDK chain and ensure that the parentCID and CID for each vmdk is intact and effectively makes a linked list of hex values that ties each delta to the next, all the way back to the parent disk.

How to look at it:
Open an SSH session to the ESX host, cd to the snapshot working directory for the VM in question (likely going to be the location of the VM’s .vmx config file).
ls -lah to check out the contents of the directory. We are looking for the *-00000X.vmdk files
Starting with the original VMDK (you can cat the .vmx file to find out where that is located), cat the .vmdk file to check out the first CID field (the parentCID for the original VMDK will be all f’s).
Take note of the CID then cat the next VMDK in the chain (i.e. the first delta’s VMDK file). The parentCID field should match the CID value you noted.
Rinse and repeat all the way up the chain.

How to fix it:
A broken link in the chain will be a VMDK that has the same parentCID as CID. Either field needs to be changed to something else (just change one letter or number is fine) then the corresponding file either up the chain or down the chain (depending on if you chainged parentCID or CID) also needs to be updated to complete the chain.
Then, run vmware-cmd .vmx removesnapshots again and it should commit correctly.

I know the above isn’t all that clear – writing from memory rarely is. Will try to update when I have an example to refer to. Note there are a lot of good examples at http://communities.vmware.com