Where was that script that ran a continuous vmotion for tests with #PowerCLI ?

— Maish Saidel-Keesing (@maishsk) January 15, 2012

The answers I got from @joerglew and @boukeg pointed me to VMJuggler written by Richard Garsthagen 

But this was an MSI file and had to be installed as an application - so I said to myself - there has to be another way to do this.

So I wrote a small PowerCLI VMjuggler (the name and credits go all to Richard)
The basic functionality in the MSI was:

1. Choose source Host
2. Choose Destination Host
3. Choose VM
4. Display counter of the number of vMotions that have taken place

###########################################################################
#
# NAME: VMjuggler.ps1
#
# AUTHOR:  Maish Saidel-Keesing
#
# COMMENT: Based on the exe file from http://www.run-virtual.com/?dl_id=8
# This will move a vm between two hosts an endless loop until the script is stopped
# It will display a counter of how many vmotions have been performed.
# VERSION HISTORY:
# 1.0 15/01/2012 - Initial release
#
###########################################################################

$hostA = Read-Host "Please enter the name of Host A"
$hostB = Read-Host "Please enter the name of Host B"
$vmname = Read-Host "Please enter the name of VM"

$x = 0
While ($true) {
Move-VM -VM $vmname -Destination $HostA | Out-Null
$hostA,$hostB =$hostB,$hostA
$x++
$x
}

Annotations:
Lines 15-17 - Accept input to populate variables
Lines 19-21 - In an endless loop move the VM between hosts
Line 22 - I would like to thank Shay Levy for explaining to me how to switch the variables. The destination for the migration is always $hostA but each time the VM moves hosts - the destination must also change and this is the method to swap the values in the variables. A detailed explanation can be found here - http://tfl09.blogspot.com/2009/02/swapping-variables-with-powershell.html
Lines 23-24 - display the counter

Below is a Demonstration video of the script.

VMjuggler demonstration

Hope you can make use of this.

Of all the features available with vSphere - one of the greatest features I like is Storage vMotion, which is described by VMware as follows:

In simple terms, vMotion allows you to move your VM from one host to another, Storage vMotion allows you to move your VM's between different Storage arrays / LUNS that presented to you ESX Host. All without downtime (ok, one or two pings.. ).

I was looking to understand more on how this exactly works - so I looked up Kit Colbert's Session from VMworld 2009 (ancient, I know but still a great source of information)

Borrowing some slides from Kit's presentation we will try and understand a bit more.

And how does this work?

That is all nice and fine - and now for a look under the covers - to see exactly what is happening.

So we start a Storage vMotion of a VM named deb1 from vsa1_vol_1 to vsa1_vol_2

The task starts running as you can see in vCenter tasks.

But the real "magic" is happening on the Host itself.

Opening a vSphere client session to the host itself, we will see a new VM that is created.

Just before the the SvMotion is completed you will see that both machines co-exist for a short amount of time and both are powered on

The switch is made and the old one is powered off and removed.

And from an esxtop perspective. Here you can see that there is one VM with an ID of 175523.

Start the SvMotion and there are two VM's.

SvMotion completes and only the new VM with its new ID (175722) remains.

And the machine is now running from vsa1_vol_2.

And that is how sVmotion works.

**Update**

After receiving a message on Twitter from Emré Celebi with the following text,

and also a comment from Duncan Epping, I realised that the information I posted was pertaining to ESX 3.5 and not 4.x.

So here is the correct technical document for 4.x.

So what changed? CBT is now used to tracks the changes between the start of the process and the last stage just before the switch over. A good explanation on CBT here by Eric Siebert.

In this great session from Ali Mashtizadeh and Emre Celebi I learnt more about the process and how it now works in 4.x.

What are the differences?

Here is the process.

So How does the Changed Block Tracking come into play?

In 4.x VMware introduced the Data Mover which can also offload the Storage operations to the Storage Vendor with VAAI.

Comparing the Old and the New

In 4.x This is the process

1. Start Storage vMotion
2. Flag the disk, and start CBT checkpoint.
3. Start pre-copy of the disk to destination in multiple iterations.
4. Check which block have changed cince the check point and copy only those remaining blocks and use Fast Suspend/Resume for the switch over.
5. Delete original.

So how does this change performance-wise as compared to 3.5? As you can see below the performance gain is substantial both in ESX CPU cycles used during the process and the time needed for the process.

Just one last thing regarding troubleshooting.

The information in the VMworld Session goes in to more detail than I have done here - so I highly advise anyone who would like to understand the process in-depth - listen/watch the full session. It is free and an hour well spent.

The screenshots posted above showing the process are from a 4.x environment so they reflect the updated method.

Thanks again to  Emré Celebi and Duncan Epping.

This one has been bothering me lately. Where do you place your Virtual Machine Swap files?

There are two options:

1. In the Same Directory  as the VM
2. In the Datastore specified by the host.

As you can see below.

If you choose the second option then you can configure the swap location per host

Here comes my question.

Duncan Epping posted about the Impact of decisions… - for those that wanted to keep the vswp files on local VMFS datastores on their Hosts. Frank Denneman also wrote a post regarding this subject as well- Impact of host local VM swap on HA and DRS.

My question is as follows. When changing this you see (in both screenshots) a nice warning from VMware that this could degrade vMotion performance.

I take this to be true if you were moving the swapfile location to a local VMFS volume located on the ESX host. When initiating a vMotion - the vswp file will have to be copied to the other host's vswp location for the migration to complete.

But what about a different Datastore that is shared storage - but not the location of the virtual machine files?

This is actually specified as a one of NetApp's best practices (NetApp and VMware vSphere Storage Best Practices TR-3749) pg. 80.

Now why would you do this? If you look on the right - you will see one reason - no snapshots, which will save precious disk space. Second reason  - no replication is needed (unless of course this is a requirement)

Some say this is a risk - because if your datastore with the vswp files goes down then all your VM's go down, which in essence is true - but… Since in most cases the datastore is just another volume on the same storage array from which the VM's are running from - the chance of the vswp datastore failing is equal to that of the chance that VM's datastore will fail.

So my question at the end of the day. Is the warning still valid when you define a shared datastore - one which all the ESX hosts in that cluster can access?

There is of course a certain overhead needed to set this option on each ESX host in the cluster - but that is not (IMHO) so much of an issue.

Looking forward to your comments.

F5 Announced today that they were providing with their BIG-IP can VMotion from one site to another without downtime.

First the VM is sVmotioned from one storage to another, and then Vmotioned, F5 BIG-IP will redirect the to the remote site, including live sessions that are currently in progress.

Taken from the Demo on their Site

In this webcast we will demonstrate how to perform a secure live migration of a VM from one cloud to another, without downtime, and without user disruption using LTM, GTM and vCenter APIs. Until now, it has been difficult if not impossible to accomplish this. VMotion is today limited to a single L2 broadcast domain, and Storage VMotions are even more restrictive. IP address space limitations make management cumbersome at best.

F5 will be demonstrating this capability on Wednesday, September 2 at 10:30 a.m. and 5:15 p.m. at the VMware vCloud Pavilion at VMworld 2009. Come visit F5 Networks at Booth #2302

Demo You can download the webcast video here

For those of you who have not heard yet, Microsoft System Center Virtual Machine Manager R2 was RTM a few days ago.

One thing I noticed on a post on the Microsoft Virtualization Blog was a reference to some of the details that are in the upcoming release (granted this is for the RC and I cannot attest to this being the same for the RTM version) and found these little points of interest.

1. I wonder where Microsoft got this "need" from? Perhaps because it is a feature that has been around in ESX 3.5 since January 2008?
2. Downtime of less than 2 minutes? (need I say more?)
3. True by default you can only perform 2 simultaneous vmotions, but this is not a hard limit and can be changed - you can see a good review of the her from Simon Long and Jason Boche in this post.
   I dread to even think if this is true - how long it will take to evacuate a Hyper-V host if they have to be done one by one. At least now you don't have wait for each one to finish before starting the other - Thanks Microsoft for fixing that one!
4. For full disclosure - granted Storage vMotion is now only available in the Enterprise Plus version of vSphere (which is not something I am very pleased about) and Microsoft claims that this is completely free (I have yet to confirm this)

On a personal note a small message to Microsoft (and their supporters).

Dear Microsoft,

You have come a long way since Hyper-V version 1. Hyper-V R2 is a great product. I personally use both - and will continue to do so. Each one has their use-cases and their appropriate time and place to be used.

There are a number of features that you still have a lot of catching up to do to get to where VMware is today, but I gather you will get there.

The competition is healthy, and welcomed and me as the customer will benefit from it without a doubt.

I personally have a favorite - but that does not mean that I will not continue to explore new and emerging technologies, be it VMware, Hyper-V, Xen or KVM

Hope you enjoyed the ride!

I had to to migrate 43 Virtual Machines today, from one Datastore to another.

The environment is still not upgraded to vSphere yet.

Options I had:

• sVmotion Plug-in
• Command line
• PowerCLI

I found the most efficient to be the following

Get-Folder "<Folder Name>" | get-vm | Move-Vm -Datastore <New_DataStoreName> -RunAsync

The plug-in was too many mouse clicks, and the command line - I have never tried - so I do not know if you can run it against multiple machines

The command completed within 3 minutes - and the storage VMotion was queued for all Virtual
Machines - it did take a bit longer

Hope you enjoyed the ride!

It is less than 1 month to VMWorld 2009 in the Moscone Center in San Francisco (August 31-September 3, 2009). A great conference to start with, and picking up more and more momentum as we draw closer.

This is the main Virtualization Event of the year, and the only one for another 14 months - seeing that VMware has decided to "consolidate" the European and US events back-to-back, which I personally think is not a good idea (but enough has been said on the blogosphere about this subject)

What I would like to discuss is what we can expect at this conference. vSphere was released less than 3 months ago - and the latest family of product, including AppSpeed, Lab Manager, and Chargeback were announced just 3 weeks ago.

Speculation from Sven Huisman about releasing vCenter for free was and interesting one that came up not long ago.

I will add some speculations of my own of one of the features that I think will be announced in the upcoming conference.

Well it is not actually a new product, but more of a feature that will be supported by at least two of the big Storage Vendors, EMC and NetApp.

I am talking about Storage DRS.

Storage vendors today are incorporate the usage of SSD (Solid State Disks) into their products. The benefits of SSD disks are not a small thing. In a presentation I saw last week from EMC, I was shown a graph of the comparison of what was needed to achieve a level of Disk I/O from a storage array. The ratio of the amount of disks Solid State:Conventional was approximately 1:10 to achieve the same performance. True - at the moment you would not save that much on pricing at the moment because SSD disk are much more expensive, but on power, cooling, space, and other overhead, I am sure you can see where the savings will come from.

DRS - is a built in feature (now only available from Enterprise Plus versions and up) that balances your virtual machines according to the load on the ESX host, allowing for overall better performance for all you VM's and your clusters. The benefits and increase (15%-47%) in performance can be seen for example in this article on VROOM. The variables that are taken into account are only RAM and CPU.

Back to Storage DRS, what if you could, on a defined schedule/policy, Storage vMotion your VM's to faster storage to allow for better performance during peak times, and when the peak was over move them back to slower storage?

Let us take for the following scenario for example. I have a storage array with 3 different kinds of disks, SATA 7200 RPM, Fiber Disks, and Solid State drives. These drives are layered at 3 different tiers: Budget, Standard and Premium (SATA, Fiber and SSD - accordingly).

VM1 runs a front-end application that needs x amount of I/O. Your client comes and tells you that during the day, between peak hours of 12.00-16.00 the application is slow. After testing and monitoring the performance, you see that during those peak hours the amount of disk I/O that this VM needs increase by 200%. and your lower end storage is the bottleneck.

You now have two options:

1. Move the VM to faster storage - and therefore solve your bottleneck problem, but in doing that you are allocating the VM faster storage which it does not need for 20 out of 24 hours in the day.
2. Use the built in tools - to set a policy that sVmotion's the VM at when the I/O to the disk becomes the bottleneck to Premium Tier and move it back again when things calm down back to Budget Tier.

In the EMC presentation I saw, this concept was presented as being a feature which is soon to be released. I received an additional hint to confirm this from Vaughn Stewart from NetApp on the last VMTN Communities Roundtable Podcast last Wednesday night. It will most probably be as a plug-in to vCenter which will allow proper integration

This is most probably one feature/announcement that we should expect during VMworld 2009 later this month. I do truly hope that the vendors will continue to develop tools like this which will enable us to provide better performance, better control, and better management of our resources in our virtual Infrastructures.

Please keep up the good work.

What other products/features do you think will be get during the conference?

I was also pointed to this post by Daniel Eason, which discusses the same subject, Thanks Daniel!

Your comments are always welcome.

In the current series of posts I am writing on running a vSphere lab on ESX 1 2 and 3, I wanted to set up an NFS shared storage between my 2 ESX hosts to test vMotion.

I ran into an interesting issue which I could hardly find any mention of on the web.

We continue our Saga. Part 2 ended with configuring our cluster and now we go onto shared storage and vMotion.
If you looked at the previous topology of my lab you will notice that there was no interface configured there for shared storage. That was a small oversight on my part which I corrected by adding a additional NIC, by the way, that is why I love working on virtual machines as a lab - hardware does not cost anything!!

I added a fourth NIC to each ESX host with a VMKernel port connected to 1.1.2.x - named NFS.
So after adding a new nic to each of my ESX hosts the topology looks like this

ESX4-1	ESX4-2

 

We will now add a shared NFS volume to each server. I know that NFS is not the most popular choice for shared storage out there, but I do have to say that I am extremely pleased with the performance, and in my personal production environment, the benefits we receive with, ease of use, backup times and administration, has made NFS the de-facto choice for all our ESX deployments.

The NFS share is hosted on an Openfiler server (well I am exaggerating a bit - it is actually a desktop with a large disk). Extremely stable - as you can see from the screen shot below.

"There is more than one way to skin a cat" - so go the saying, and there is more than one way to add an NFS volume.

We can do it through the VI client.

 

 

Or we can do it from the command-line on the ESX host.

esxcfg-nas -a <volume name> -o <hostname/ip> -s <share name>

or in my case

 

And now we have both Servers which see the same Shared Storage.

And this is a diagram of my environment

I now fired up a Windows Server machine to test out my vMotion

 

Next up on the menu Fault Tolerance