Be Careful of the 9th Host in the Cluster

I was looking yesterday at post from Matt McSpirit regarding the Cluster Limits of Hyper-V. I started to look at how this would compare to vSphere.

I do not want to go into who can cram more VM's into a cluster VMware / Hyper-V? I did notice something that I realized - I think I should point out.

Take the following Scenario.


    Now you want to add another Host to your cluster.

    On pg. 7 of The Configuration Maximums for vSphere 4.0 and vSphere 4.0 Update 1 you will see that when adding the 9th host - the number of VM's per host is limited to 40.

    So therefore the scenario will change to this:


    In thinking that you can increase your capacity in your Cluster but adding another host - you just reduced the number of VM's that can run in your cluster 280!! In order to get back to the same amount of VM's you were previously running - will need another 7 additional hosts

    As always my advice is - read the documentation and research the implications of your actions will be.


    Get-AllDomainControllers Function

    Sometimes you would like to run a query against all DC’s in your forest. For example today – I was looking a specific error on the System Event Logs.

    Here is a quick way to get all the DC's in the forest adapted from Marcus Oh’s blog.

    # ==============================================================================================
    # NAME: Get-AllDomainControllers
    # AUTHOR: Maish Saidel-Keesing
    # DATE  : 22/06/2010
    # COMMENT: 	Will collect all Domain controllers, set the list as the $alldcs variable
    #			and output as list 
    # ==============================================================================================
    function Get-AllDomainControllers{
    	$myDomain = [System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain()
    	$script:alldcs = $myDomain.Forest.Domains | ForEach-Object { $_.DomainControllers } | `
    		Select-Object -Property Name | Sort-Object -Property Name
    ### Entry point to script

    From there to get the event I wanted was easy

    $alldcs | ForEach-Object {
    Get-EventLog -LogName System -ComputerName $_.Name -source KDC -EntryType Error -Newest 10 | where {$_.EventID -eq 27} |  select-object TimeGenerated,MachineName,message


    Benchmarking your Disk I/O

    How fast is your storage? How do you even check what kind of throughput you are getting?

    That is a question that comes up often enough, when performing benchmarks.

    I actually dealt with such a case last week. The customer was performing a storage Migration of all their VM's from an HP MSA Array to a NetApp Filer. In order to ensure that no decrease in performance was experienced, I needed to benchmark the the disk I/O before and after the migration.

    There is an "unofficial storage performance thread"  which is on the VMTN forums. This is actually an follow up on a even older thread that started way back in February 2007.

    The principle is the same.

    1. Prepare a Windows VM. This VM does not need anything specific in it, so a basic install will do just fine.
    2. Add a Second Disk to the VM (10GB will do fine)
    3. Format the new Drive with NTFS.
    4. Download Iometer - according  to your operating system (32/64 bit)
    5. Download the Unofficial performance config test file
    6. Extract the zip file. Copy the Dynamo.exe and IOmeter.exe files (located under the .\iometer-2008-06-22-rc2.win.32\iometer-2008-06-22-rc2\src\Release folder) to your VM
    7. Copy the OpenPerformanceTest.icf file you downloaded in Step 5 to the VM.
    8. Double click on the IOmeter file.
    9. You will be presented with a window like this below

    10. Click on the folder icon to load the config file for the test and point it to the OpenPerformanceTest.icf 
    11. Choose your newly created drive that formatted in Step 3
    12. Click the Access Specifications Tab and add the bottom four tests to the Assigned Access Specifications
      You can if you would like click on edit to see what each of these specifications will test - and customize to your liking if need be.
    13. Click the green flag to start the test.
    14. You will be asked for the to save the file.
    15. IOmeter will then begin to prepare the drive.
      What does that mean. IOmeter will create a file in the root of the drive you selected (approximately
      4 GB, that is why 10GB is more than enough, and then perform the tests on that file).

      This can take a while but let the process run.

    16. Once the preparation is complete the tests will run, each test for 5 minutes. During that time you can see the real-time results by choosing the results Display tab

    17. You can see which test is currently running at that time by clicking the Access Specifications Tab

    18. Once the tests are finished (20 minutes) you will be presented with a CSV file.
    19. Open that file in Excel.
      The interesting columns that you would like to use (if you want to compare to others in the thread) are:
      G, J, O and AT (IOps, MBps, Ave Response Time, % CPU Utilization)
      • IOps - The Average # of IOps achieved / sec
      • MBps - Throughput in MBps
      • Ave. Response Time - Average response time in ms
      • % CPU Utilization - How much of the vCPU was utilized in %.
    20. I usually run the test 3 times - and collect the average of the three.
    21. The next time you run the test - you will not have to wait to prepare the drives be cause the file is already there.
    22. To clear the results click on reset workers button

    23. Start again at Step 9.

    Share your findings with others in the community by posting your results to the
    unofficial storage performance thread

    Update(25-Jun-10): As a result of he discussions based on the comments of this post and with the generous help of Didier Pironet of DeinosCloud - I am adding below a more "real-life" configuration test file for IOmeter.

    Didier prepared this Config file according to HOWTO - Benchmark an iSCSI target (and any block target, including within VMs living in NFS datastores) from Chad Sakacc.

    IOmeter Benchmark

    The specs are pretty clear and divided in 4 groups: Regular NTFS, Exchange, SQL and backup/restore

    4K; 100% Read; 0% random (Regular NTFS Workload 1)
    4K; 75% Read; 0% random (Regular NTFS Workload 2)
    4K; 50% Read; 0% random (Regular NTFS Workload 3)
    4K; 25% Read; 0% random (Regular NTFS Workload 4)
    4K; 0% Read; 0% random (Regular NTFS Workload 5)

    8K; 100% Read; 0% random (Exchange Workload 1)
    8K; 75% Read; 0% random (Exchange Workload 2)
    8K; 50% Read; 0% random (Exchange Workload 3)
    8K; 25% Read; 0% random (Exchange Workload 4)
    8K; 0% Read; 0% random (Exchange Workload 5)
    8K; 50% Read; 50% random (Exchange Workload 6)

    64K; 100% Read; 100% sequential (SQL Workload 1)
    64K; 100% Write: 100% sequential (SQL Workload 2)
    64K; 100% Read; 100% Random (SQL Workload 3)
    64K; 100% Write: 100% Random (SQL Workload 4)

    256K; 100% Read; 100% sequential (Backup)
    256K; 100% Write100% sequential (Restore)

    Those two specs below don't mimic any Operating Systems/Applications behavior. It just tells you the maximum IOPS the storage can supply.

    512B: 100% Read: 0% random (Max Read IOPS)
    512B: 100% Write: 0% random (Max Write IOPS)

    Virtualization Domain Controllers - #2

    I received a number of comments on my previous post and offline as well I want to clear up some things regarding the previous post.

    I am not against virtualizing domain controllers - not at all. I am against trying to P2V a domain controller. It seems that was not clear enough from the last post. I do still advise that if your risk analysis - you should do that before virtualizing your domain controllers (or anything else for that matter) comes out that it is more cost effective to keep one Physical DC then do so. There are a number of reasons to go either way - it all depends on your environment and what you are willing to risk.

    It will save you so much headache and anguish - if you would just promote a new clean VM to a Domain controller.

    Now lets go into a a good reason (or two) why you should actually virtualize your domain controllers

    1. Domain Controllers do not need that many resources

      Of course there is a whole science behind this and a good amount planning guides out there - you can see that a domain controller does not need that many resources. It has relatively stable RAM usage (loading the AD database into RAM is mostly what it does) - CPU usage will depend on the authentication traffic - same with Network and Disk IO.
      The resource usage can be predicted very well - so you can plan the resources for such a Machine.
    2. You should have more than one - if you don't then you are not doing your job correctly. But that means if your one of your Domain controllers fail - because your ESX host failed - then you are still up and running - because you have another DC running (either on an ESX host or physical).
    3. Testing purposes. You need to prepare your AD Schema for the upgrade to Exchange 2010. Even though the schema extension is a Microsoft product, and has probably been tested countless amount of times with all levels of AD Domains. But of course every domain is different. And no matter how many times Microsoft have tested it - it has not been tested on my domain, in my environment, with my applications.
      So for this I would need to test the upgrade. So what better way to do it on a replica of my Production Environment.
      All I need need to do is to power off the DC, copy the VM to a closed lab environment, power them both back on, fix up a whole bunch of stuff to get it working in the lab without access to the outside world, and hey I have a full replica of my production domain that I can test.

    So as you can see there are benefits to virtualizing your DC's - just a few of them above.

    What other benefits would you add to the above list? I would appreciate your comments.


    VMworld Sessions - Close but no Cigar!

    After waiting most the day, and seeing a number of people on Twitter who announced that they had sessions that were accepted, I was waiting for an email - and yes, lo and behold - it arrived this evening - along with many emails to others, the rejection letter.

    Last night on the Communities Roundtable - the VMworld team was there, including Richard Garsthagen. He spoke about the 1000 Sessions submitted for the call for papers this year. About how many were accepted, about the fact the that the number of votes was not the only factor for having a session accepted.

    I posted a short while back - which were the popular session that were up for voting. This is not testify as to which sessions received the most votes, but to which track that people found to be most popular to submit a presentation for.

    Am I disappointed? Yes - I am. I actually did think that one (of my three) sessions would be accepted this year. But I guess it was not meant to be.

    I have my reservations and opinions about the way the whole process was handled (public voting process, timetables etc.) but I will take that offline.

    I would like to thank all 50 of you who voted for my vCenter Migration - It's a Snap with POWERCLI session - Thanks for the vote of confidence - it is highly appreciated!!

    It is my nature to always try to see the good side of things in events, so here is what find to be the bright side of none my sessions being accepted:

    1. First and foremost I will have more time to prepare for my VCDX Design Application. It is a huge amount of work, and I do not feel that I would have done it justice if I would have submitted before the June 21st deadline. I now have some more time to complete the work properly for a defense in Copenhagen before August 2nd.
      As of now I am not sure if I will be attending VMworld in San Francisco - but I will be in Copenhagen.
    2. It will clear off my schedule a bit for the upcoming period to start working on a new project that I will be able to disclose in the not too far future.
    3. This will also allow me to blog a bit more - something that I have been finding hard to find time for, in the last couple of weeks

    There are exciting things up ahead, many, many things to blog about and so much knowledge to share.

    Congratulations to all of you had their sessions accepted - I am sure you will all do a great job!!

    I do plan to still prepare a presentation for each of my sessions that I submitted. I would be happy to share them here with you all on the blog when they are done.

    Updating Pictures for Users in AD

    I came across a nice article last week regarding pictures for User accounts in an Active Directory environment

    First let describe the issue at hand. Every now and again, my manager gets an email saying,
    "Why don't I see the picture of xxx when I open up his mail?"


    And shortly thereafter I get an email asking the exact same question.

    Well the answer is always give is that up until Exchange 2003 - there was no option of adding a picture to GAL (Global Address List) - and even if you did you had to jump through hoops to get that picture to appear in Outlook on every computer.

    Starting with Outlook 2010 - that feature is built-in so it makes life a lot easier.

    Now of course this does not only work with Outlook - but with everything that uses this GAL - for example OCS, SharePoint ….

    So how do you get the picture into the GAL? Simple put the picture into the Users attribute in Active Directory.. :)

    Uhmmm …. Simple? How do you do that?

    With Powershell it is, which leads me to the article I came across on Mike Pfeiffer's Blog which shows you how. He is using the native AD Cmdlets - but that of course only works if you have a Windows 2008 Domain Controller or the Active Directory Web Services installed in your domain.

    And if not?

    I used the Quest Active Directory Powershell Cmdlets to do the job.

    #Convert the image to Bytes
    $photo = [byte[]](Get-Content C:\temp\maish.jpg -Encoding byte)
    #Set the Attribute on the User
    Set-QADObject -Identity -ObjectAttributes @{thumbnailPhoto=$photo}

    Next to come up - what is the size of the Thumbnail that should be put there?

    Recommended is no larger than 96x96 pixels and is limited to 10Kb in size, anything bigger will not work.

    How does this affect the Active Directory Database?

    Well adding in more information will make the DB grow - but for 1000 Users x 10KB you get 10MB so that addition is negligible IMHO - User certificates that are stored in AD are about the same size. Add to the fact that it is a one time synch - because most of this data is static anyway.

    You will have make sure that the attribute is replicated to the Global catalog in the Schema


    Now the only part hat needs to be taken care of - is getting a proper image from HR in the correct size. Maybe we will find a Powershell script to take care of this :)


    Virtualizing Domain Controllers

    One of the frequent questions that come up on the forums is,
    "How do I convert (P2V) my Windows Domain Controller (or SBS Server)?"

    Let me first start with the following statement.


    Now that I have that off my chest - lets explain why and provide some references to back that up.

    A Domain Controller could possibly be - and probably is - one of the most important computers on your network. Almost everything relies on Active Directory:

    • Authentication
    • Mail
    • Web
    • File Access
    • etc. etc.

    If your Domain controller is not functioning - then rest assured - slowly but surely a lot of other things will stop working shortly thereafter.

    I was reading a good blog post from the Active Directory Team on their blog  - How to Virtualize Active Directory Domain Controllers (Part 1). This is a Hyper-V centric article - but it is relevant to VMware as well .I do advise giving the full article a good read.

    This is what I have taken with me from the above article.

    1. In most environments there is no reason not to virtualize your Domain Controllers. If they are only being used as Domain Controllers (and not File Servers, DHCP, Web Servers) then unless you have a very large or extremely busy environment your DC's will not need an extravagant amount of resources so it can run very nicely as a virtual machine.
    2. I would always, ALWAYS, leave at at least one physical server running as a domain controller on the network. The reason being, if your virtual infrastructure depends on your Active Directory infrastructure - and it always does then if your DC's are not available due to your Virtual Infrastructure being down, or your storage being down, then you will have a serious chicken and egg situation - with not being able to easily bring up the storage or the Virtual infrastructure because they are dependant on DNS / Active Directory and that cannot come up because the the storage / Virtual infrastructure is not available.
      Jason Boche posted an article last week describing a situation where he had a network component fail - which brought down his NFS storage. One of the VM's on that storage was his Domain Controller. Once the failure was fixed, he could not bring up the NFS datastores - because they were relying on Name Resolution - and the DC was a VM on the NFS datastore, which could not be mounted, because there was not Domain Controller. As I said Chicken and Egg. True there are ways to get around it but I sleep better at night spending that extra amount of money on a physical server for a domain controller.
    3. Time Synchronization. A  domain controller should always be synchronized with an external time source. do not rely on the internal VMware Tools.
    4. Do not stop or suspend Domain controllers. Leave them on or power them off.
    5. Do not restore a Domain Controller from a snapshot. You will run into USN Rollback problems.
    6. Back up you Domain Controller the same way you would back up a Physical Server, be it NTBACKUP, Windows Backup and Restore (for Windows 2008 and Up) or a 3rd party backup client.
    7. If you try to P2V  a DC you will most likely run into a USN Rollback problem - Knowledge Base Article.
    8. The Easiest way to migrate a Domain Controller is to install a new VM, DCpromo the VM as a new DC and then remove the old one. The process of migrating the data in Active Directory from one computer to another is really simple and completely taken care of by Windows, so do not try and complicate things.
    9. VMware KB 1006996 - Virtualizing existing domain controllers.


    New Updates of VMware Products

    As Always, Test Before you deploy!!!

    VMware Data Recovery 1.2 Available.

    Release Notes - Download Link - Admin Guide

    Here are some highlights of the new release:

    • a file level restore client for Linux virtual machines
    • ability to run up to 10 VDR appliances per vCenter Server instance
    • ability to fast switch between the deployed appliances via the vSphere Client plug-in
    • Miscellaneous vSphere Client Plug-In user interface enhancements including:
      • The means to name backup jobs during their creation.
      • Additional information about the current status of destination disks including the disk's health and the degree of space savings provided by the de-duplication store's optimizations.
      • Information about the datastore from which virtual disks are backed up.


    VMware vCenter Server 4.0 Update 2

    Release Notes - Download Link

    Here Are some Highlights of the new release

    Guest Operating System Customization Improvements: vCenter Server now supports customization of the following guest operating systems:


    VMware ESX 4.0 Update 2

    Release Notes  -  Download Link

    Here Are some Highlights of the new release

    • Enablement of Fault Tolerance Functionality for Intel Xeon 56xx Series processors— vSphere 4.0 Update 1 supports the Intel Xeon 56xx Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel Xeon 56xx Series processors.
    • Enablement of Fault Tolerance Functionality for Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors— vSphere 4.0 Update 1 supports the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors.
    • Enablement of IOMMU Functionality for AMD Opteron 61xx and 41xx Series processors— vSphere 4.0 Update 1 supports the AMD Opteron 61xx and 41xx Series processors without input/output memory management unit (IOMMU). vSphere 4.0 Update 2 enables IOMMU functionality for the AMD Opteron 61xx and 41xx Series processors.
    • Enhancement of the esxtop/resxtop utility vSphere 4.0 Update 2 includes an enhancement of the performance monitoring utilities, esxtop and resxtop. The esxtop/resxtop utilities now provide visibility into the performance of NFS datastores in that they display the following statistics for NFS datastores: Reads/s, writes/s, MBreads/s, MBwrtn/s, cmds/s, GAVG/s(guest latency).
    • Additional Guest Operating System Support— ESX/ESXi 4.0 Update 2 adds support for Ubuntu 10.04. For a complete list of supported guest operating systems with this release, see the VMware Compatibility Guide.
    • Resolved Issues In addition, this release delivers a number of bug fixes that have been documented in the Resolved Issues section.


    VMware Tools Icon Changed – Did you notice?

    While installing VMware tools on a virtual machine this week on VMware Workstation 7.1
    (build-261024) I noticed that the small icon that appears to show that VMware tools are installed looked different.




    image image

    Will it be the same for the next update of ESX??


    VMware vExpert Awards 2010

    Yesterday the vExperts for 2010 were announced.

    VMware awards the vExpert to individuals who have significantly contributed to the community of VMware users over the past year. vExperts are book authors, bloggers, VMUG leaders, tool builders, and other IT professionals who share their knowledge and passion with others. These vExperts have gone above and beyond their day jobs to share their technical expertise and communicate the value of VMware and virtualization to their colleagues and community.

    I am happy to announce that I am one of those who have received this award, and I am grateful to have been granted such an imagehonor.

    I have created a Twitter List that I will be updating - with those who have received the award.

    If you are not on this list - please let me know - so I can add you.

    Congratulations to all those that got the first round - there will be another round of emails going out on Monday.