2010-09-29

Host Crash because of ESX Active Directory Authentication

I was subscribed to this thread and received an update today that I must have missed previously, with an issue much more serious than the one that the thread was originally started with.

Previously, I posted an article to explain how to Integrate ESXi into Active Directory. After the integration logging onto a host with your domain credentials can cause the host to crash. The issue occurs when the user logging into the ESX host with their Active Directory credentials is a member of more than 30 Security Groups.

The issue has been confirmed with happening with the user being a member of as little as 23 security groups as well.

The ESX Host throws an error and reboots (PSOD) and reboots. The thread owner has a ticket open with VMware and according to the information there engineering are working on the issue.

Below are Screenshots taken from the thread

imageimage

I ran a quick check to how many groups my AD account is a member of (Powershell of course)

Get-QADUser msaidelk | Get-QADMemberOf -ErrorAction Continue | where {$_.GroupType -eq "Security" } | Measure-Object | Select-Object -Property Count | Format-List

Count : 45

I have tried to re-create this on my standalone host but have not experienced this problem.

Has anyone else encountered this issue?

2010-09-28

No. 32 - Thank you all so much!

The list of the top 25 blogs was released yesterday.

I unfortunately did not make the top 25 this year - close but not enough. I am honored to be placed anywhere on this list, be it number 1, number 32 or even 115. The way I look at it is that people are enjoying the content I write on my blog and think it is useful, and the ranking I received - is for me a great vote of confidence and makes me extremely proud!

Sure there are benefits of being a top 25 Blogger - be it advertising opportunities on your blog, exposure and there others I am sure (and perhaps also the prospect of being hired by VMware/EMC/Veeam/NetApp). But we do not do it for those reasons.

I blog partly for my own benefit of having a place of reference to document things that I come across during the day, but also because I feel that I have people can benefit from the knowledge that I have to share. I think that I can say for certain that every single one of the top 25 bloggers do it for the same reason. We do not get paid for this - we do it because we enjoy it.

I would like to make a few observations I had on the results.

  1. I am happy to see that there are at least two new scripting bloggers that have been added to the list. This is becoming more and more important as environments grow larger and larger. Congratulations both to William Lam and to Luc Dekens on their places in the top 25. Very well deserved!
  2. You do not have to blog every day with something new to become a popular blog. There are some bloggers on the top 25 list that have not posted more than 15 posts in the last 3 months. Which leads me to believe that people did not really read the criteria that Eric posted when the voting opened, and did make this into some kind of popularity contest.
  3. There are several top bloggers that have moved under the employment of a major vendor over the past 6 months. It seems that the amount of time that these people now have left to blog during their day is not as much as they had before, and therefore there blog updates have suffered both in quality and in quantity, and they have slipped down in the list.
  4. The amount of votes - 860 - was extremely low (IMHO). Last week I went out for a drink with Frank Denneman and Kenneth van Ditmarsch - they both brought up the subject which makes you wonder.

    If you take into account the number of unique views per day that each of use see on our blogs
    (I know for example Duncan Epping has over 5,000 / day and I am sure that Chad Sakacc has a good number per day as well), that seems to be a very low number. Either our statistics are wonky - or the public is not that interested in participating in voting (but elections have never been a really popular subject..)

Congratulations to all the top 25 bloggers!!

You can follow Eric Siebert's new feed of the top 25 blogs

I have updated my Top 25 bloggers Twitter list.

This as always gives me more motivation to continue to think of new ideas to provide good content for the blog. Thank you to all of those who voted and especially to those who voted for me.

2010-09-21

Murphy's Law

Let me introduce you to a good old soul by the name of Edward Murphy. Who was Edward Murphy?
In short he was responsible for the famous saying, "Anything that can go wrong, will go wrong".

And so the story goes.

After preparing with the rest of my team members about what should be done - how it should be done, and going over all the current open issues that needed to be taken care of while I would be away on vacation, I thought, "Great! Finally some quiet and relaxing times while I go on vacation".
But our dear old soul Murphy? Nuhuh, he had other plans …

I received an email 6 hours after I had flown out of the country - saying that a the following VM's were deleted.

Deleted? Deleted? How? What? When? Why???????????????????????

So what happened? User Error. Plain and simple. Someone on the storage team wanted to delete the snapshots on an NFS Qtree but by mistake, instead of deleting the snapshots they deleted the volume. And in the flick of a button, Boom! 40 VM's were gone!

image
Source (Deepspar)

A great deal of SMS's alerts, email notifications and several phone calls later, the error was identified.

Now we were left with 40 orphaned VM's in the infrastructure.

By the way - there is no undo button here.

So what did I get from this experience?

  1. My boss has already told me that the next time I can take vacation - is somewhere in 2020 :)
  2. No matter how much redundancy you plan for (Raid Groups, Storage Processors, Disks, Network), there is always the unknown things that catch you with your pants down.
  3. Human Error - is a good percentage of the reason for outages. Mistakes can be made, mistakes will be made. You can cover 99% of the cases - but again it will always be that 1% that will get you.
  4. Backup - Backup - BACKUP.
  5. Restoring from Backup can take a while. Quite a while

Back to my vacation….

2010-09-17

vCloud - Enterprise as well not only Enterprise Plus

After having a Twitter conversation with Massimo Re Ferre' - VMware vCloud Architect,
(his article is a great read - you should definitely make the time to get to understand it -
vCloud Director Networking for Dummies)
he was surprised that the whole conversation was going on because as far as he knew the Ent. Plus was not a pre-requisite for vCloud Director (even though the information states so on the VMware website)

These are the updates that he posted today.

RT @maishsk: @mreferre Any news regarding the Licensing and cloud? <- yes. Ent is supported. Web will be updated.Thu Sep 16 17:46:23 via TweetDeck

@maishsk @hany_michael @jasonboche @langonej to recap ... I got confirmed Ent is supported (along with Ent Plus) with vCDFri Sep 17 08:06:44 via TweetDeck

*** Updated 22/09/2010 ***

The information on the Website has now been changed to reflect that Enterprise is supported, but Enterprise Plus is Strongly recommended

image

Well I guess this closes this previous post VMware vCloud Director - Enterprise (Plus) only? 

I am really pleased to see that the proper people are paying attention to the feedback coming from the community are are quick to react on it. Thank you.

Now it will be interesting to hear what happens regarding - The Future of VMware Lab Manager

2010-09-14

Vote for the Top 25 Virtualization Blogs

It is that time of the year again, and it time to perform your duties and help me stay in the top 25 bloggers list. Eric Siebert has opened the survey on his site.

Top 25 Bloggers

 

The list is a large, 100 blogs, but you can only pick your top 10.

I hope you enjoy the content I provide and will put me in the top 3!
(Seriously though - anywhere on this list will make me happy).

I am honored to be listed amongst such a great list of people. 

So how does it work. 3 stages

1. Select your top 10 favorite blogs, you will be asked to rank them in the next question. Current top 25 blogs are listed first followed by the remaining blogs in alphabetical order. You can view links to all the blogs on the vLaunchpad. You must select exactly 10 blogs to continue.

2. Rank the 10 blogs that you clicked in the previous question with #1 being your most favorite blog. Rankings are weighted with a #1 vote equal to 10 points and working down to a #10 vote being equal to 1 points. Here you will have to drag and drop your selections, move them around until you get the order correct

3. Enter your name, email and a Captcha Code.

Who knows, you might even win some prizes.

  • Two random voters will be picked to win a copy of Eric's book, Maximum vSphere or a copy of the Train Signal vSphere Pro Series Vol. 2 video training course.
  • 2010-09-13

    User Can't Cancel a task

    I was asked today to cancel a task of a machine import today - because it was created by mistake. So I naturally asked the Cluster Admin, "Why do you not do it yourself?". And he naturally answered me, "Because I can't! The option is greyed out."

    Now there are certain tasks you cannot cancel in vCenter. The exact list - I am sure LucD will be able to pull this out of the SDK - but this was not the purpose of this post.

    The way permissions are set up in my environment - is least amount of privileges needed is what is granted.

    Each Admin is assigned Full Administrator permissions on their Cluster with Propagation set. On the Top level folder is whole different ball game.

    The way I do my daily work is by assigning myself the same rights as the Cluster Admin (msaidelk)- and all Elevated privileges are performed with another user (admin) that have full control over the whole Environment. I do my PowerCLI work with that elevated user.

    So I checked with my msaidelk user and I also could not cancel the task.

    image

    So I checked if it was because the task was not cancellable. That was simple with PowerCLI

    get-task -Status running | select ID, StartTime, IsCancelable| fl


    Which gave me output that the task was cancellable.

    Id           : Task-task-33529
    StartTime    : 13/09/2010 18:59:40
    IsCancelable : True

    I tried to stop the task with my admin user

    Get-Task | where {$_.id -eq "Task-task-33529" } | stop-task -confirm:$false

    image

    image

    Worked fine. So this had to be a permissions issue.

    I looked on the Permissions for the Top folder and found my issue

    Global -> Cancel Task was not checked

    image

    You might have thought that if the user was an admin on the cluster then they would be able to cancel the task but I guess not.

    Added that right to role and Voila!

    image

    image

    2010-09-12

    Converter 4.3 - Throttling

    In my previous post about Converter 4.3 - My First Impressions - I mentioned a new feature called Throttling which was introduced in the new version.

    So what does it do? You can limit the amount of resources that will be used by the Conversion Task.

    I would like to clarify two things about throttling:

    1. This is only valid for Windows Machines this has no effect for Linux.
    2. The throttling is not done one the ESX host.

    You can use throttling in two different scenario's

    1. Conversion of a Powered on Windows OS
      In this case the throttling occurs on the source machine being converted - minimizing the resources used by the conversion process.
    2. Conversion of a Powered off Windows OS
      In this case the throttling occurs on the machine where you are running the converter application - minimizing the resources used by the conversion process as to not interfere with the regular operation of the OS.

    Why would you want to do use throttling? Because a conversion task competes for computing resources with other processes. To limit the impact of the conversion task on other workloads, you can throttle the task. If you do not throttle conversion tasks, they are carried out with normal priority and depending on the available computing resources during conversion.

    Lets give an example.

    Case 1 - Powered on OS - The throttling is performed on BIG_SERVER

    image

    Case 2 - Powered off OS - The Throttling will occur on SMALL_CLIENT

    image

     

    The CPU throttling drop-down menu, has 3 options

    None - The priority of the current conversion task is set to normal.

    Light - The priority of the current conversion task is reduced slightly below normal.

    Medium - The priority of the current conversion task is set to lowest.

    You can also throttle the maximum network bandwidth that will be used during the conversion

    You cannot throttle disk I/O from the Converter Standalone interface, but by changing the network bandwidth indirectly will have an effect on the Disk I/O

     

    It will be interesting to see numbers on what the difference is.. From a capture of a conversion, I did not see the process changed to a lower priority

    image

    2010-09-06

    Converter 4.3 - My First Impressions

    There are a few changes to note with the new Converter version.

    You can now import Hyper-V Machines Directly

    image

    There is a new Throttling feature (More about that in an upcoming post)

    image

    You can now define a cluster as a destination for your conversion - I guess DRS will sort out where to place the VM.

    image

    As opposed to

    clip_image001

    And Last But not Least. It is blazing fast. Give it a spin…

    Great Job VMware!!

    2010-09-05

    VMware vCloud Director - Enterprise (Plus) only?

    Please see the update in the follow up to this post.

    OK VMworld is now over. I was actually not in San Francisco, but with all the updates from Twitter, the Keynotes, and #thecube I felt part of the conference. From all the feedback that I have heard from the show, it seems that it was a great event, and extremely beneficial.

    I will not go over all the product releases that were announced over the past week, there have been more than enough posted last week.

    I would like to go into a bit more detail about the vCloud offering that VMware has now made available.

    Up till now I have been asked - a number of times

    • What are we doing for cloud in our organization?
    • How are we building the cloud?
    • What are we going to use for Cloud?

    Cloud has been a very nice buzzword used at all levels - but is not really understood properly at any of the levels.

    Those of you who have used in the past VMware Lab Manager - will recognize some similarities between the two products, be it the Libraries, Organizations and so on.

    I would like to stress a few things

    1. I have not yet taken the product for a test drive yet (but I will)
    2. I would like to recommend Hany Michael's great tutorial on Taking vCloud Director for a Spin which provides a great introduction on how to get this up and running on your laptop/desktop with only 8GB of RAM.
    3. The only database that you can use is Oracle. At present there is no support for MS SQL in this release (It will be added soon)
    4. And last but least - this only works with dvSwitches.

    For me the last item is the one that stands out most for me and is the most problematic. Let me explain why.

    Approximately 15 months ago VMware released vSphere 4.0 and with it they decided to change their Licensing Model - and introduced a new Licensing level for some of the new features - Enterprise Plus. Originally the plan was to retire Enterprise edition completely, but due to several reasons - one of them I suppose was pressure from the customers - they decided to ditch that plan.

    But - 2 Main features remained that were available only in Enterprise Plus - Host Profiles and Distributed Virtual Switches. NIOC and SIOC were added with the 4.1 release.

    We have all been waiting for VMware to release a product to manage the cloud and until now my answers to all the question above were, "We are preparing - but as of today there is no product to actually manage it properly". This one seems to provide what it says. Till now this has been a concept only, with vCloud this is now a reality. We can create our own Private clouds - move the machines between our clouds - private or public. So this will be a product for everyone and from now on forward to deploy their VM's in the cloud.

    But - And this is a big BUT….

    vCloud Director will not work without dvSwitches (**Please see update Below**) - which means (at least from the way understand it) you have to have Enterprise Plus on all of your ESX hosts that are managed under the system. Yep.. Every Single one of them. But to be completely accurate - only those host in clusters that you will use as resources for vCloud.

    I do not think that everyone will make use of this product - for the simple reason of licensing. Ian Koenig mentioned the fact that

    Lab Manager was now dead and that the use case for the product would no longer be needed, now that we have vCloud Director.

    I disagree - vCloud will be used and I think you would be stupid not to use it if you have the licensing already in place. But this is aimed at the high-end customers. Perhaps this will change in future but today - this is a high-end / enterprise level product only. Lab Manager will have its use cases as well.

    But before we all go around and deploy our own clouds - remember - it something that you have to pay for - and it is not cheap..

    Is this a bad thing? I do not think so. But time will tell.

    *** Update ***

    Duncan Epping has pointed out to me that vCloud Director will work on a regular vSwitch - which would make this completely a licensing issue and not a technical one.

    I did go over the documentation that was released and found mention almost exclusively of only dvSwitch, not a regular vSwitch except in one place.

    Add a network pool that is backed by port groups to register vSphere port groups for Cloud Director to use. Unlike other types of network pools, a network pool that is backed by port groups does not require a vNetwork distributed switch.

    So I retract my statement about the technical requirement of dvSwitches for vCloud Director.

    You still need Enterprise Plus Licenses for the product regardless. That does not change.