Hot Add and "Need have have"

One of the new features that were added to ESX4 is Hot-Add. What this feature does it allows you to add additional RAM or vCPU to your VM (if the operating System supports it)

Like any good vAdmin - I have upgraded my personal ESX box to Version 4. This is a whitebox with ESX3i installed.

Upgrade process was pretty simple as you can see in this previous post.

So if you noticed in my previous post after installing your license is a 60-day Evaluation with all the bells and whistles like below

The option of ESX4i - Free version - is available for up to 999 CPU's all you have to do is just ask.


So I requested a license key and after receiving the mail from VMware I proceeded to enter my license. Now we all knew that when entering the free license key in ESX3i you lost some functionality - like VCB - Virtual Center Agent etc. But for ESX 4i when changing the license - You lose a lot more in comparison - and I was shocked to find out how much.

clip_image001[4] clip_image001[6]

One of the wonderful things you can do with ESX VM's is resize the VMDK's - it is a simple

vmkfstools -X 30G /<…..path to …. >.vmdk

And voila your vmdk just grew to the size you asked it to. So now you boot up your VM (for arguments sake - let's say it is a Windows 2003 Server) and you now have the extra space but in this case this was the system volume which cannot be extended while the system is live.

So the solution to this one is :

  • Extend the volume
  • Add the vmdk to a running Windows 2003 Server (while it is powered on)
  • diskpart
  • rescan
  • select disk 1 (of course the number may vary)
  • select partition 1 (same as above)
  • extend
  • Power off of this VM
  • Remove the disk
  • Power on the first VM again and hey presto you have a full partition including the extended space you just created.

Now this is all fine and wonderful - and it all relies on the fact that we expect that we can add resources to a running machine (namely HD / NIC - those are the ones that I use the most). This can be used in a large amount of scenarios - such as the one I had today - I had a lock on a file that I just could not get rid of. So I tried to do the same thing:

  • Power off the VM
  • Add the VMDK to another running machine
  • remove the lock
  • but …. hold on I could not !

I ran into this error:


ESX claims that I am trying to hot add a resource - which in all honesty is completely true.
I was. A vmdk.

So i wanted to check was this connected to the serial number I just changed before hand. So of course changing the license back to evaluation version I could add the hard disk without any issue. So I changed the license back and tried the same with adding a network card.

And yep you guessed correctly - same error message.

  1. The error message of course is not very well phrased - and I would love to know what "need have have" is?
  2. Why in heaven's name would you take away the most basic functionality that has been in every single VMware product (besides Player) for the past 5 years? VMware? Please tell me that was a mistake?

Update 30/05/2009

I have started a thread on the VMTN forum with this question but have as yet to hear any concrete response from VMware. The only shocking thing I found that if this is a licensing issue - then we are all in for a bigger surprise! The minimum, license level that you use Hot-Add is Advanced.

How do you like them apples?


Veeam FastSCP and ESX4i

One of my valuable tools that I use is Veeam's FastSCP. A while back they introduced a new version that was compatible with ESX3i - which was a great blessing!

Today I tried to connect to my ESX4i server and I found that it would not

Received an error:


I started to look on their support forum, but could not find any mention of ESX4i - so I reverted to Twitter.

I posted:

maishsk @veeam - does fastscp work with ESX4i - I am getting an unknown api version 4.0 error http://twitpic.com/5zq83
Tue, May 26 17:43:39 from TwInbox

and lo and behold (10 minutes minus a few seconds later)

veeam @maishsk Currently testing version 3.1 with the GA code of vSphere 4.0, should just be a few more days. Veeam Backup first, then FastSCP
Tue, May 26 17:53:46 from digsby in reply to maishsk

Twitter of course is an amazing tool, and I don't really how people work without it. Don't you love it?

So just to let you all know FastSCP will not work in its current version with ESX4i - but will very soon.

Update: I was using an older version to be exact. I downloaded a newer version 3.0.1 today and all works fine. Thanks again to Veeam for wonderful support and follow up. Well done!

Virtual Thoughts

No,  I have not started to become philosophical and all, this is about a new podcast that will be held this evening.

The forum will be non-VMware centric, non-security centric podcast on virtualization technologies.

Virtual Thoughts will tackle several items to start, like Licensing, Performance, Open Source tools, Cross Platform products, etc. This is not a product show but a bringing together of a discussion about these and other pressing issues, and questions in the industry. Hopefully with some sort of solid take away that either the vendors or the customers can apply.

So as you can see from the above – this will not be only a VMware podcast – for that we have the Communities Roundtable, and it will not be only a security podcast, for that we have Edward’s Security Roundtable. Most of the panelists use VMware’s products extensively – but not exclusively

Among the Panelists on the Podcast will be experienced members of the Virtualization community (in no particular order):

The Podcast will be weekly occurrence and you can access it here on Talkshoe.

Looking forward to it.


ESX3i to ESX4i Update

Released with the new version on Thursday was a upgrade package from ESX3i to ESX4i

After you have downloaded the package – proceed to update your host.

The example below is a VM installed with latest build of ESX3i (158874)


Open the VMware vSphere Host Update Utility (this will not work with the VI 2.5 client – it must be the vSphere 4.0 version) and select the host you would like to upgrade

Add the Upgrade file. The file is verified

Accept the EULA and enter Credentials

You will have to have put the host in Maintenance mode beforehand otherwise you will receive an error like this

Compatibility checks are run – in my case this did not work the firs time round because the VM only had 1GB of Ram allocated so I got this. Duncan has written an informative post on the matter and how to manipulate the amount of RAM if you need to.

So after adding additional RAM to the VM the compatibilty check passed

And The Upgrade Process Begins

And after waiting a while the Host is updated with the new version of ESX

A Few points to note:

  1. Do not close the Host Update Utility during the upgrade otherwise you will most probably mess up your host
  2. You of course need the minimum amount of memory (2GB) for the upgrade to run
  3. When the host comes back online it will be in evaluation mode – you will have to provide a license for it.(you have 60 days)
  4. Of course your host has to be in Maintenance Mode before you upgrade.

Happy Upgrading!


The vSphere 4.0 Beta – Thanks!

Many of the bloggers that are active in the community, participated in the vSphere 4.0 Beta, I was amongst the chosen few.

We were all under NDA, sworn to secrecy and not allowed to publish any info as to what the world was in store for. Later on down the road some information started to leak out, questions about vSphere and the NDA arose, but let bygones be bygones.

Knowing something that others don’t is kind of fun – but at times can also be frustrating – I mean you want to discuss your opinions, experiences and disappointments with others – but you actually can’t – because you are under NDA.

The amount of improvements and new features that were released with the new version are countless – so many new things, so many things improved – I am confident that this is a better product, has more functionality and will empower us to all present better solutions to our customers, our clients and our employers.

The Beta Forums have been full with posts, information and problems – and most (if not all) issues were dealt with accordingly in the forums by the VMware staff and the other Beta testers.

Now the cat is out of the Bag – the release is publicly available – and all the information is now public knowledge – for the good and the bad. And we all have a lot of work to do.

Expect a large amount of information coming from the Blogosphere in the upcoming future (if you haven’t already noticed)

Below is a small ”Thank You” note from VMware. I think we all deserve it.


vSphere is born

Well not really born, but it went GA now.

After a decent amount of Twitter traffic going back and forward first regarding the knowledge base that started to show up with vSphere KB’s and documentation that was posted

Download Link is Live

The Rush is on!!!

As long as the servers are not running under Hyper-V they won’t crash ….

vSphere HCL – Downloadable

As of now there is only an online HCL.

Till the official documents are released – here are the documents in PDF

Systems Guide

I/O Devices



VMware Online Virtualization Forum

The Agenda has been released for this event.

Attend the Online Virtualization Forum on June 9th and 10th,
8:00am–1:00pm PDT. Don't miss this opportunity to:

  • Network with VMware product experts, customers, partners and other IT Professionals.
  • Attend 30-minute presentations and get the latest info on VMware solutions, products and industry trends.
  • Access real-time information from industry leaders without the hassle or expense of travel.
  • Create a multi-media resource kit, including white papers, videos and podcasts.

It is Free Virtual event(online)

The Agenda and the sessions that caught my interest



Maybe I will see you there?


Kodiak 0.0.3 Released

I received an email this morning that Bluebear have released a new version of the management.

Here are the changes from the release notes

Kodiak 0.0.3 Release Notes:
Thanks for downloading Kodiak!

This version of Kodiak represents a significant change in our server-communication architecture.

  1. We've added a schema-compliant request/argument builder engine that populates requests based on known information. This makes building requests much easier.
  2. A lua scripting engine. Each server connection gets its own independent Lua script.
  3. Independent object pools.
  4. Improved mapping response and control.
  5. Map connection node & link display control.
  6. We've created a "plugin api" where a user may build a plugin to communicate with any kind
    of data service. We'll be publishing information on this on our wiki.
  7. We've added a number of "boilerplate" features that will become more apparent in the next
    few releases as we push Kodiak towards our goal of creating an "IT IDE".
  8. Access for users to create their own scripts. Kodiak's UI/inspector controls are driven by a "global schema" that allows a user to create their own customized scripts for management. Please visit the bluebear wiki for further information regarding customizing the script engine. As yet, we don't have a full UI for managing user scripts, but we will shortly.
  9. The remote event log has been removed for this release.
  10. For Windows users who experience the dreaded "SSL certificate bug", we've added a preferences option to bypass the windows SSL stack. This option is accessible from the Preferences menu.
  11. Password vault & credentials storage. Kodiak now has the ability to save username/password pairs in an encrypted password vault, locked with a master password.
  12. Server object creation is disabled while we migrate the control components into the VMWare plugin, we will re-enable these in a couple of days.

I have tried the previous releases of the product out, and find it to be an interesting idea. I personally had severe issues with it running on a windows machine because of SSL certificate issues (which it seems they have addressed), so I moved to running from a Ubuntu host instead.

One of the better changes that I see now is the documentation page, which I found was non-existent till now, so I hope this will improve  my experience with the product and enable better use.

I have a few invitations that I can give out so if you would like one, leave a comment on this post

A proper review of the release will come out soon.


Enable EVC on a Cluster

One of the wonderful useful features that was released with ESX 3.5 U2 was Enhanced VMotion Compatibility (EVC).

From the release notes:

Enhanced VMotion Compatibility – Simplifies VMotion compatibility issues across CPU generations. Enhanced VMotion compatibility (EVC) automatically configure server CPUs with Intel FlexMigration or AMD-V Extended Migration technologies to be compatible with older servers. After EVC is enabled for a cluster in the VirtualCenter inventory, all hosts in that cluster are configured to ensure CPU compatibility for VMotion. VirtualCenter does not permit the addition of hosts that cannot be automatically configured to be compatible with those already in the EVC cluster.

Why am I blogging about this now, well I came across a situation this week, with a cluster that we wanted to enable EVC on. This is how the story goes.

Client wanted to upgrade RAM on two 100% identical hosts (same server, same CPU, same configuration, same BIOS level – everything). In order to bring them down he needed to VMotion the VM’s off, and upgrade the RAM. For some reason some VM’s (mainly RHEL 64bit) would not Vmotion off the host, and we were getting CPU mismatch errors.

We powered down the problematic VM’s and cold migrated them to a new host. Upgraded the RAM, and at the same time, BIOS upgrades as well. Brought the one host back up – migrated the VM’s again (cold migrations for the problematic ones) and brought down the second host. RAM upgrade, BIOS as well, and brought the host up again.

Vmotion between the hosts worked fine. But in order to prevent such issues in the future, he wanted to enable EVC. So you cannot enable EVC on a cluster that currently has hosts inside of it,



As is detailed in this post – what you need to do essentially is:

  • Put host in maintenance mode
  • Remove host from original cluster
  • Create new EVC enabled cluster
  • Add host to it (check NX/XD and Intel_VT options in system BIOS. These options need to be enabled)
  • VMotion files from old to new cluster.
  • Disarm old cluster

So I started the process, and when I tried to Vmotion a VM from a non-EVC into the new EVC-enabled cluster I got this error


Now going to KB 1993 gives a wealth of information about “VMotion CPU Compatibility - Migrations Prevented Due to CPU Mismatch - How to Override Masks” but in order to change the masking for the VM – you need to power it down (which if I already was going to power the VM off, then I would already Vmotion it into the new cluster), or enable a global setting in the vpxd.cfg file (the info is in the KB). This I was not going to do because this is not the only host in cluster in the Datacenter.

We rechecked that both VT and check NX/XD options were enabled in the bios, and that the output of

cat /proc/vmware/cpuinfo

on the ESX hosts were identical.

Carter Shanklin suggested to perhaps, suspend the VM’s, and then try to Vmotion them, but that did not help either.

So I guess I will have to revert to Shutting down all the VM’s, Vmotion them into the new EVC-enabled cluster and then power them on again.

What do you think could be the cause of this kind of behavior? It should work seeing that both hosts are identical.

Unfortunately, I do not have another solution at this time.


Error Converting to Template

Today I came across an issue that seemed to be strange.
I tried to convert a  template to VM in order to apply some patch updates.
When trying I got a error message:



So I went into the folder where the template was located to see if I could see anything there.
What I found was that there for some reason was that there already a vmx file in the folder which was causing the convert process to fail.


[root@ /vmfs/volumes/Templates/templates/windows/T_Win2k3_SP2-32b]# ls -la
total 15736624
drwxr-xr-x 1 root root 2048 Feb 11 14:03 .
drwxr-xr-x 1 root root 80 Nov 11 21:59 ..
-rw------- 1 root root 16106127360 Jan 21 12:27 T_Win2k3_SP2-32b-flat.vmdk
-rw------- 1 root root 8684 Jan 21 12:27 T_Win2k3_SP2-32b.nvram
-rw------- 1 root root 409 May 10 16:31 T_Win2k3_SP2-32b.vmdk
-rw------- 1 root root 457 Jan 1 10:55 T_Win2k3_SP2-32b.vmsd
-rwxr-xr-x 1 root root 2815 May 10 16:15 T_Win2k3_SP2-32b.vmtx
-rwxr-xr-x 1 root root 2813 Feb 11 14:03 T_Win2k3_SP2-32b.vmx
-rw------- 1 root root 271 May 10 16:15 T_Win2k3_SP2-32b.vmxf
-rw-r--r-- 1 root root 21712 Nov 11 15:47 vmware-14.log
-rw-r--r-- 1 root root 18716 Nov 11 15:47 vmware-15.log
-rw-r--r-- 1 root root 28084 Nov 11 15:47 vmware-16.log
-rw-r--r-- 1 root root 22793 Nov 11 15:47 vmware-17.log
-rw-r--r-- 1 root root 46379 Nov 11 15:47 vmware-18.log
-rw-r--r-- 1 root root 39589 Nov 11 15:47 vmware-19.log
-rw-r--r-- 1 root root 40180 Jan 21 12:27 vmware.log

[root@ /vmfs/volumes/Templates/templates/windows/T_Win2k3_SP2-32b]# mv T_Win2k3_SP2-32b.vmx T_Win2k3_SP2-32b.old.vmx

I renamed the file and did the same command again.




Question now is how did the file get there in the first place???


A short vSphere Presentation

I presented an overview of some of the new features that are upcoming in vSphere to my colleagues today.

vSphere Pricing Comparison

I wanted to put up this post mainly for myself, but I sure you can find this useful when trying to compare the differences between VI3 and vSphere. There have been a numerous amount of technical details and improvements with the new version. but in this post I want to address the price differences

VI3 Pricing and vSphere Pricing. All prices include 1yr Gold Support.

licensing comparison

Here is what I will take with me from this comparison.

  • Standard license just a hell of a lot cheaper..
  • If you do not need Storage VMotion and DRS – then you can use the new Advanced License instead of the old Enterprise and save some major $$$.
  • The Essentials package is a huge upgrade on my view of what you got in Foundation.
  • Starting from December 16, 2009 – Enterprise will become more expensive.
  • ESXi – will still be offered as a free version with per incident support ($299) / SnS Gold/Plat ($495/595).


I Would Love a Rolls-Royce but …

With the Virtualization world buzzing and with latest Microsoft announcement from today that the Windows 2008 R2 Hyper-V Server will be completely free of charge (and will come with Live Migration and HA) and the prices and licensing of vSphere

A few questions have come up:

  1. Will this change completely take over the SMB market with a product like this?
  2. Because of the above what will happen to the enterprise market?

Here is how I see it. If you look at the table that was provided on the post above then the direction in which Microsoft is going is becoming clear.

They presented this table:


Now if we compare any vSphere product that has HA and VMotion – the lowest level possible would be Advanced so the prices seem right (and we will not mention the fact that there is no VC license in here). I am not going into features and everything other reason to continue using VMware.

For the SMB market – the additional feature set is nice but, and especially in times like these, the bottom line will be $$$. And unfortunately for VMware in this battle they are not coming out on top 
– not by a long shot….

But as always this is the nature of competition and the open market – as soon as another player comes into the equation – the competition between them grows – and in the end – we all benefit, because prices drop.

My bottom line.. VMware is the Rolls-Royce of Virtualization – and anyone that says different needs their head examined. The new kid on the block (and yes believe me they are here) is Microsoft, and Citrix Xensource in my eyes is becoming less and less of a major player.

Being a Rolls is nice – but not everyone can afford it – and those that cannot – will look for something that will save them money – and sometimes at the expense of functionality. Does this change my view of what is the better technology – no it does not, but this is not the purpose of the post.

So for all those folks in Palo Alto – I think it is time to start looking over your shoulder, the competition is creeping up on you…