2012-03-22

Quick and Dirty #PowerCLI to Repair a Disk Space Problem

I ran into a case of a datastore that ran out disk space (why that happened is a whole different story – don’t go there… really….) but because of a whole strange chain of event this caused a number of VM’s to become corrupted.

So how did they become corrupt? There were a number of VM’s that were in the middle of committing a snapshot and there was no space on the volume. The VM’s became corrupt. I could not power them on, I couldn’t do anything with them. The only thing I could do was to restore them from backup - and that is what I did. (There were a number of other VM’s on this datastore as well – but they froze and were waiting for a prompt)

I was asked – how many VM’s were affected by this and which VM’s – and what I did want to share with you is - how I found the list of machines to restore.

First step was to get all the events that happened between 22.00-24.00 

$a = Get-VIEvent -Start (Get-Date).Addhours(-14) -Finish (Get-Date).Addhours(-12) -MaxSamples 10000 


[12:00:07 PM] ~> (Get-Date).Addhours(-14) 
Wednesday, March 21, 2012 22:00:13 PM 

[12:00:13 PM] ~> (Get-Date).Addhours(-12) 
Thursday, March 22, 2012 00:00:19 AM
I then looked for the events that happened on a machine that I knew had been affected
$a | ? {$_.ObjectName -like "MACHINE1*"}

EventTypeId          : com.vmware.vc.VmDiskFailedToConsolidateEvent
Severity             :                                                       
Message              :
Arguments            :
ObjectId             : vm-1346
ObjectType           : VirtualMachine
ObjectName           : MACHINE1
Fault                :
Key                  : 29721794
ChainId              : 29716968
CreatedTime          : 21/03/2012 22:56:59 PM
UserName             : NDR-IL\vi3admin
Datacenter           : VMware.Vim.DatacenterEventArgument
ComputeResource      : VMware.Vim.ComputeResourceEventArgument
Host                 : VMware.Vim.HostEventArgument
Vm                   : VMware.Vim.VmEventArgument
Ds                   :
Net                  :
Dvs                  :
FullFormattedMessage : event.com.vmware.vc.VmDiskFailedToConsolidateEvent.fullFormat
                                      (com.vmware.vc.VmDiskFailedToConsolidateEvent)
ChangeTag            :
DynamicType          :
DynamicProperty      :

One of the events was the one above. Now that we have found an EventTypeId that looks like the problematic one – I looked for all the machines that had this error. (VM Names of course have been altered to protect the innocent…) 

$a | ? {$_.EventTypeId -eq "com.vmware.vc.VmDiskFailedToConsolidateEvent"} | select CreatedTime, ObjectName, FullFormattedMessage | ft
CreatedTime                    ObjectName      FullFormattedMessage
-----------                           ----------              --------------------
21/03/2012 23:21:08 PM  MACHINE1        event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 23:06:10 PM  MACHINE2        event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 23:04:34 PM  MACHINE3        event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 23:03:38 PM  MACHINE4        event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 22:57:17 PM  MACHINE5        event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 22:57:17 PM  MACHINE6        event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 22:57:04 PM  MACHINE7        event.com.vmware.vc.VmDiskFailedToCo...

Get the list of VM’s and on which datastore they were located so that I start a restore….

$a | ? {$_.EventTypeId -eq "com.vmware.vc.VmDiskFailedToConsolidateEvent"} | select ObjectName | sort ObjectName | % { 
$x = get-vm $_.ObjectName 
  Write-Host "$($x.Name) - $($x.ExtensionData.Config.Files.VmPathName)" 
}
MACHINE1 - [NFS_2] MACHINE1/MACHINE1.vmx
MACHINE2 - [NFS_3] MACHINE2/MACHINE2.vmx
MACHINE3 - [NFS_2] MACHINE3/MACHINE3.vmx
MACHINE4 - [NFS_3] MACHINE4/MACHINE4.vmx
MACHINE5 - [NFS_2] MACHINE5/MACHINE5.vmx
MACHINE6 - [NFS_2] MACHINE6/MACHINE6.vmx
MACHINE7 - [NFS_3] MACHINE7/MACHINE7.vmx

Quick and dirty (and I am sure that it is not really my best coding) – but Oh So Useful!!!!

You see why you have to learn PowerCLI ??

Labels: , , , ,