Converting a Linux Machine fails during /var clone

I was having some difficulty with the conversion of a Linux machine. Every time I tried to import the machine it would fail at about 19%.

So two things I wanted share with you today.

  1. If you are using the integrated vCenter Converter (which you should.. ) then you can monitor the results of the conversion from the vSphere client itself.
    If you right click on the task you will get a view summary option. Here you can monitor the conversion and get the history of prior attempts as well.

  2. This is what I was getting during the conversion


    So Mr. Google was my friend as usual (or at least I thought so) which brought me to KB1026348:
    Redhat 4 conversion fails or the virtual machine does not boot after successful conversion

    I tried step 4. which worked fine. there were no errors in the /var/log/messages file either.

    I admit this had me stumped for a while. I have done several of these conversions (today as well) without any issue before so Enable Retention of Sparse Files During Linux Conversions (1008303) was not appealing - it required a restart of the service and RedHat's tar bug fix update did sound right either.

    So after searching a bit more I came across this thread. Now my solution was not exactly the one that was presented there but in my case it was an abnormal amount of files in  the var partition.

    I had found 148371 files that were located in /var/spool/clientmqueue/. It seemed that sendmail had barfed all over itself and had gone whacko.

    After removing all those files - I was able to import without an issue.



Storage vMotion - A Deep-Dive

Of all the features available with vSphere - one of the greatest features I like is Storage vMotion, which is described by VMware as follows:


In simple terms, vMotion allows you to move your VM from one host to another, Storage vMotion allows you to move your VM's between different Storage arrays / LUNS that presented to you ESX Host. All without downtime (ok, one or two pings.. ).

Updated info vSphere 4.x here

I was looking to understand more on how this exactly works - so I looked up Kit Colbert's Session from VMworld 2009 (ancient, I know but still a great source of information)

Borrowing some slides from Kit's presentation we will try and understand a bit more.


And how does this work?




That is all nice and fine - and now for a look under the covers - to see exactly what is happening.

So we start a Storage vMotion of a VM named deb1 from vsa1_vol_1 to vsa1_vol_2

Before Migration

The task starts running as you can see in vCenter tasks.

Start Task

But the real "magic" is happening on the Host itself.

Opening a vSphere client session to the host itself, we will see a new VM that is created.

New Machine_1

New Machine_2

Just before the the SvMotion is completed you will see that both machines co-exist for a short amount of time and both are powered on

New Machine_3

The switch is made and the old one is powered off and removed.

New Machine_4

And from an esxtop perspective. Here you can see that there is one VM with an ID of 175523.


Start the SvMotion and there are two VM's.


SvMotion completes and only the new VM with its new ID (175722) remains.


And the machine is now running from vsa1_vol_2.

Migration complete

And that is how sVmotion works.


After receiving a message on Twitter from Emré Celebi with the following text,


and also a comment from Duncan Epping, I realised that the information I posted was pertaining to ESX 3.5 and not 4.x.

So here is the correct technical document for 4.x.

So what changed? CBT is now used to tracks the changes between the start of the process and the last stage just before the switch over. A good explanation on CBT here by Eric Siebert.

In this great session from Ali Mashtizadeh and Emre Celebi I learnt more about the process and how it now works in 4.x.

VMworld Session

What are the differences?


Here is the process.


So How does the Changed Block Tracking come into play?


In 4.x VMware introduced the Data Mover which can also offload the Storage operations to the Storage Vendor with VAAI.




Comparing the Old and the New

Old - 3.5 New - 4.x

In 4.x This is the process

  1. Start Storage vMotion
  2. Flag the disk, and start CBT checkpoint.
  3. Start pre-copy of the disk to destination in multiple iterations.
  4. Check which block have changed cince the check point and copy only those remaining blocks and use Fast Suspend/Resume for the switch over.
  5. Delete original.

So how does this change performance-wise as compared to 3.5? As you can see below the performance gain is substantial both in ESX CPU cycles used during the process and the time needed for the process.


Just one last thing regarding troubleshooting.


The information in the VMworld Session goes in to more detail than I have done here - so I highly advise anyone who would like to understand the process in-depth - listen/watch the full session. It is free and an hour well spent.

The screenshots posted above showing the process are from a 4.x environment so they reflect the updated method.

Thanks again to  Emré Celebi and Duncan Epping.


Update Intel NIC Drivers on ESX 4.1

I Installed an IBM x3650 M3 the other day. During the installation the additional Intel NIC was not recognized by default in the ESX Host.

This I could see in two different ways, from the output on the console

msaidelk@esx9:~$ sudo lspci | grep Ethernet
0b:00.0 Ethernet controller: Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T (rev 20)
0b:00.1 Ethernet controller: Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T (rev 20)
10:00.0 Ethernet controller: Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T (rev 20)
10:00.1 Ethernet controller: Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T (rev 20)
15:00.0 Ethernet controller: Intel Corporation Unknown device 1516 (rev 01)
15:00.1 Ethernet controller: Intel Corporation Unknown device 1516 (rev 01)
1f:00.0 Ethernet controller: Intel Corporation Unknown device 1516 (rev 01)
1f:00.1 Ethernet controller: Intel Corporation Unknown device 1516 (rev 01)

And the GUI also only recognized the first 4 Broadcom NICs (instead of 8)


I posted an article about IBM x3650 M3 Does not Recognize NICs a while back - but as you can see from the output above they are recognized in hardware - just ESX does not know how to deal with them.

I downloaded the driver from VMware's Site and extracted the files from the ISO image and the file I am interested in is in the offline-bundle folder


The host has to be in Maintenance mode for the patch update.

Install through vCLI

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>vihostupdate.pl --server esx9.maishsk.local --username root  -i -b \\vc\VMware\ESX\INT-intel-lad-ddk-igb-2.4.10-offline_bundle-320657.zip

After installation

msaidelk@esx9:~$ sudo lspci | grep Ethernet
0b:00.0 Ethernet controller: Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T (rev 20)
0b:00.1 Ethernet controller: Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T (rev 20)
10:00.0 Ethernet controller: Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T (rev 20)
10:00.1 Ethernet controller: Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T (rev 20)
15:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
15:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
1f:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
1f:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)

Host rebooted and it comes up with all NICs recognized.


I wanted to try this PowerCLI as well. I ran into some issues that I will address in a different post.


VMware Horizon App Manager

VMware has launched Project Horizon !!

Here is the VMware Horizon App Manager landing page

DatasheetFAQ  -  News Release

Q. What Is VMware Horizon App Manager?

A. VMware Horizon App Manager is a hosted service that enables organizations to centrally manage the provisioning, access and usage of software-as-a-service (SaaS) applications. With this turnkey solution, IT departments can extend on-premises Microsoft Active Directory identity to the public cloud, simplifying the security of application access. In addition, strong policy management delivers robust user activity reporting. End users, even those with multiple devices, have a single login and simplified, self-service access to the organization’s application store.

Q. How does VMware Horizon App Manager work?

A. VMware Horizon App Manager provides a cloud identity service layer for securing managed access to SaaS and Web-based applications. Specifically, IT managers will utilize a Web-based SaaS application management tool to provision and entitle access to end users. The identity and access management (IAM) technology in VMware Horizon App Manager unifies silos of user identities in the private and public cloud into a single cloud identity, increasing the security of SaaS and Web-based application usage for the enterprise. In addition, managers will have control over user access policies and be able to track SaaS license activity centrally, via usage reports.

This is not a downloadable product - for more information please visit this form.

Why Should you Care about Veeam Support for Hyper-V

I will not go over the details of this announcement that was released today.
Both David Davis and Sean Clark wrote very good articles about the release.

I do want to add a small point of my own, and that is why I think this is a big thing - or at least the start of a big thing.

There is no doubt that:

  • VMware is the current market leader.
  • Microsoft see VMware as the biggest competitor in the virtualization market.
  • vSphere is the more mature product (and depends who you ask, more robust as well).

After this announcement it is clear - Hyper-V is here to stay.

I am a VMware fan (if you haven't noticed then you are not following my blog) but part of my job is to provide the best solution to my customer - and that solution depends on the customer. If the solution that suits them best is not based on VMware technology - then so be it - because it is right for the customer.

A year or two ago - the vendor bashing and FUD from both sides was in abundance, Microsoft said you don't need this feature - and VMware said you cannot live without it. VMware said that Microsoft does provide this - and Microsoft played catch up with the technology race to get those features that VMware has. Bad mouthing - from both sides - but I am happy to say - that it has calmed down as of late - and both parties are focusing on what they have - and not on what the competitor's do not.

So back to Veeam's announcement and why is this a big thing (IMHO)? Veeam is a company that has based 100% of their product (and revenue) on VMware. From day one. I think that it has paid off and they have found a very good place in the market for all of their products. And now the announcement that they will be adding support for Hyper-V. If a company that has based their complete business on VMware - are now opening up to different Hypervisors - then is a turning point. Not only is Microsoft getting part of the customer share - and how much is not the issue - they have started to get into the surrounding companies that make their bread and butter off of virtualization. I have - for a while - been asking any vendor that I have tried their technology, what are your plans for multi-platform support. Be it monitoring solutions, backup solutions capacity management or cloud - and I hearing always the same answer - at present we do not - but it is on the roadmap! Some for Hyper-V some for XEN, some for RHEV and some for all of the above. Veeam is not alone - the ISV's realize that they cannot afford not think about supporting multiple vendors and this is a good thing especially for Microsoft

Veeam is going the extra mile here by developing features that do not even exist for Hyper-V, as you can see the slide below


I will not be going into the debate of what the better/faster/cheaper/sexier product is. I do have to agree though, Microsoft is gaining market share - both with the customer and with the ISV's as well.

I would be happy to hear any comments.


10 Things That are on my Mind

I was recently asked what are the things that I currently working on and what is occupying my time.

I thought I would share that list with you - as it holds a list of things that you should have on your list of things to do:

  1. How do I provide more automation for my environment? Mainly for the end user.
  2. Securing my environment – performing security audits, reviews – what tools can be utilized.
  3. Updating documentation of the current environment.
  4. Defining standards.
    • What should be virtualized? What not?
    • If not - then Why?
    • If yes – how do we present insight into how resources are really used, and who uses them?
  5. Capacity planning and analysis.
  6. Preparing for the internal cloud (if you are not already there).
  7. Site redundancy.
  8. Backup and restore of VM’s – should they be treated the same as your physical machines? If not what are the tools we should be looking at to provide the optimal process to restore in different scenarios.
  9. Keeping an eye on what the competition have – and when (if at all) is worth my while to switch to a different vendor. (be it hypervisor/storage/network etc.)
  10. Preparing for the upcoming release of certain Hypervisor.

Do you have anything else you want would like to add? Feel free to leave a comment below.


VMworld Session Voting is now Open!!

Sorry for the shameless plug – but the amount of amazing sessions that are available are enormous –
1164 of them to be precise – and all of these are open for public voting.

Forbes Guthrie, Scott Lowe and yours truly have submitted a session as an opportunity to present your questions and voice your opinions on vSphere Design at VMworld. Tom Howarth has kindly agreed to moderate the session.

Session 1843
Your vSphere Design and a Session with the Experts

Every infrastructure administrator would like to have the best of all worlds from their virtual infrastructure. You want a solid and sound foundation to build your datacenter and your cloud environment for the future. Join in this session with the 3 authors of VMware vSphere Design who have "written the book" on how to get the most out your environment. Come join us for a one hour panel session where you can quiz the Experts on your Virtual Infrastructure Design. Join the Virtualization Experts and Authors; Forbes Guthrie from vReference, Scott Lowe of EMC, and Maish Saidel-Keesing from Technodrone for chance to get the answers to your vSphere design problems.

I think that such a session will be beneficial to all that participate.

If you think so as well – then come on over and cast your vote on the Public Session Voting. You will need a VMworld account – which is easy to set up.

From the FAQ:

Q: Who can participate in Session Voting?
A: Anyone with a
vmworld.com account can vote.

Q: How long is Session Voting open?
A: Session Voting is open May 9 – May 18.

Q: Am I voting on sessions for both events – US and EMEA?
A: Yes, this is global and 80% of sessions selected will occur at both events.

Q: I’d like to vote on a specific track and/or session. How can I do this?
A: Once you login to the voting site, select a Track and a dropdown menu will appear with the relevant topics. You can also search by Area of Interest (Healthcare, SMB, etc.) or Keywords (Session ID, title, speaker, etc.).

Q: How many sessions can I vote on?
A: You can vote on as many sessions as you’d like. You can only cast one vote per session.

Q: How do I cast my vote?
A: Click on the “Thumbs Up” symbol next to the Session ID. You will receive confirmation that your vote has been counted. To remove a vote, you will need to click on the on the “Thumbs Up” symbol again and this will remove the vote.

Q: Will my votes be saved?
A: Yes, your votes will be saved and you will be able to view them each time you login to the voting site.

What are the technical level descriptions?
A: There are three technical levels:

  • Business Solution – Sessions in this category will cover business benefits and real world examples. These sessions are suitable for any audience interested in the benefits of using certain technologies within their organization.
  • Technical – Sessions in this category will cover technical aspects and share best practices. Targeted to anyone with a technical background, but with little to no previous knowledge about the specific topic.
  • Advanced Technical – Sessions in this category will dive deep into the technology and will assume that the audience has previous knowledge and experience with the specific topic.

Q: Who should I contact if I have any additional questions?
A: You can send an email to