I have stated before (here), vCenter Orchestrator was the least utilized and least understood product in the VMware product suites, which is a shame – because the capabilities in Orchestrator are pretty much endless.

I started to do some number crunching and the results are quite interesting (although obvious).

Let us have a look at the number of threads in the PowerCLI community vs. the Orchestrator community over the past 4 years

You can see of course that Orchestrator is far behind and that PowerCLI is the most popular over the years.

Let’s have a look at a different perspective – by the total number of threads per year

Again PowerCLI seems to be the most popular.

How about how many posts were there in each of the communities on average per month?

Obvious results again.

Some of my analysis on the data above.

1. The number of threads that people posted in the forums does not necessarily correlate to the number of people who actually use the product – but I think it is safe to assume that the number would represent the actual use in the field.
2. vCenter Orchestrator 4.0 was released in February 2008. The number of people who used it in the following 3 years was minimal – and an in my opinion that is because the documentation was horrible. There was no definitive guides, not enough information and and not enough VMware people pushing the product.
3. That changed in March 2012 – and why was that – well I think because of the Automating vSphere: With VMware vCenter Orchestrator book by Cody Bunch. This was the first decent piece of literature that allowed people to try the product.
4. If we would compare the ratio of amount of questions asked every month between the PowerCLI and Orchestrator communities from 2010 to 2014 you would see that it would be (approximately) 10:1 in 2010 and 4 years later it is now 2:1. That is a great leap – and if you ask me this gap will get smaller and smaller.
5. I think the reason is – we have evolved, matured. 4 years ago most of the people who were managing VMware environments were the System Administrators, who were extremely comfortable with Windows and PowerShell – so PowerCLI was a natural and sure bet.
   Javascript?? No ways – wouldn’t touch it with a barge pole. But today – people have become more open to other tools, have found that PowerCLI cannot do everything that they want, environments are becoming more complex and the regular Powershell script just does not cut it any more.

I am extremely happy that people are finally getting better use of vCenter Orchestrator. I was suspecting this was the case – and now we finally have the numbers to back this up.

There are 3 very strong VMware employees that are monitoring the Orchestrator community -
Christophe Decanini, Burke Azbill and Jörg Lew – all of them really know their stuff! (I would advise that also follow them all on Twitter if you are not already)

For the PowerCLI community – the one and only Luc Dekens (the man who eats, breathes and sleeps in the community),  Robert van den Nieuwendijk and Hal Rottenberg (all of them should be in your Twitter follow list). As opposed to the Orchestrator community – none of these three are VMware employees, which (in my honest opinion) shows their devotion and dedication – and their continued contribution to the PowerCLI community.

The one thing that does come here to mind – is that the other virtualization vendors or the other solutions – do have quite a way to go until they are at the level which VMware has achieved.

I will leave you all with these final words…

With last week's release - there was also an updated version on PowerCLI that was released as well.

VMware vSphere PowerCLI 5.5 Release 1 Release Notes

I noticed something quite strange (personally I have not yet hit this bug - but I am sure it must have happened to someone)

That sounds like fun… :)

So I put a question up on the the PowerCLI community.

What exactly does this mean?
Will it run the first 2?
Last 2?
Even lines?
Odd lines?
Just choose which some random lines?

I received the answer from Dimitar and thought that it would be worthwhile to share it with you all.

What this means is that it will either run the whole script or just the first line. We haven't been able to work out exactly what causes this, but we believe it has something to do with the way we escape special symbols and redirect the output in the script before sending it to the guest OS.

This is not a new issue for this release, but we decided to add it to the release notes. There are a couple of workarounds for this
1) make the script single line
2) save the script in a .bat file, upload it to the guest using Copy-VMGuestFile and execute the file using Invoke-VMScript.

Something you might want to remember…

You have been tasked with a task, it could be a long term project, a one-off thing. These usually involve identification of number of tasks and stages that need to be executed in order to complete the whole task.

One such an example that I would like to discuss today is the completion of a complex scripting task.

One script I have been nurturing is a deployment script for Oracle RAC on VMware. It actually has grown to be over 700 lines of code (perhaps one day I will be able to make it public).

I was asked not so long a go, “How do you manage to write such a long script? How or where do you even start?”. The answer to that question is the reason for this post.

I really love working in PowerShell and PowerCLI – that is no secret. I do dabble in other scripting languages as well but Powershell is still my favorite.

Scripting can be used in different ways.

• One time quick and dirty tasks. You pop out a line of code which gets the specific information you want, and perhaps you will save it (you should) – just in case you need to do it again. These are also referred to as one-liners.
• Functions – here you think a little more, I want to create a piece of code that can be used on a regular basis – make it re-usable and write it to be robust, mean and lean.
• A process or a workflow. Now I know some of you will say that Powershell/PowerCLI is not really the ultimate tool to use for running complex tasks with multiple scenarios and use cases – and I must say I agree. On the other hand – vCenter Orchestrator has a steep learning curve. Using a tool that you already know and are familiar with will make it easier. Easier to write, optimize, troubleshoot, and document.

No matter which tool you use the methodology will be the same. This is how I do it – and perhaps this can help you too.

1. Envision the process from a high level perspective
2. Layout the steps you need to perform in order to get there
3. For each step – detail what exactly needs to happen
4. Write the code you need to get it done.

It is best to explain this by walking through an example (without actually writing the code)

Each VM that is created is usually done by an admin/user. You would like to have some way of knowing who created the VM, when and for whom (owner/department).

So how would you go about doing this?

Step 1 – Envision the process

This is how I see this – from a very high level.

Someone creates a VM, the details are filled in, and a report is sent out. Look at it from the perspective of upper management – they do not care about the details, how it works, they look at the process.

That was the easy part.

Step 2 - Layout the steps

When a VM is created it is logged, and these logs can be parsed or checked for specific events. From that information you will extract the information needed (Date, Created) and attach that info to the VM, the easiest way would be to add it to a custom field for the VM. This process will be performed for each of the new VM's that were found.

All of this information should should be collected into a readable format that can be sent as a report, either a file or an HTML page.

Step 3 – Describe each step in detail

VM Created

A VM is born. How do you know that this happens? Well pretty simple. Go through the events that were created in vCenter. But you have to remember of course that there are several ways to create a VM and these could be:

1. Deploy from OVF
2. Deploy from Template
3. Create From Scratch
4. Clone from Existing VM

Not all of these have the same events registered in the Events and Task so you will have to look for all of them to catch all VM's created.

You really do not want to scan all the events (from time immemorial) so you should decide on a timeframe - an this will be the period you will check against.

So if I were to layout my steps they would be as follows.

## Define Period to search - 24 hours
## Get All events for VM created
    ## Cloned VM's
    ## Deployed from OVF
    ## Created from Scratch
    ## Cloned From Existing

But in order to even begin to get events I would need a few things (seeing that I will be running this as a scheduled script):

1. PowerCLI
2. Connection to vCenter

So my steps will change a bit to look like the following:

### Validate connection to vCenter ###

## Check for PowerCLI
## Check for Connection
    ## If not connected then connect
        ## Need vCenter Name and credentials

### Get All VM's Created ###

## Define Period to search - 24 hours
## Get All events for VM created and store in a variable
    ## Cloned VM's
    ## Deployed from OVF
    ## Created from Scratch
    ## Cloned From Existing

OK So now I have all the VM's created within the past 24 hours. Here would be my next steps.

Update Fields

From the event details of each event for each VM - get who the user was that created it and when.
Populate that into a Custom field/Tag for each VM.

So if I were to layout my steps they would be as follows.

### Update details for each VM

## Go through each event
## Update Created By
    ## Extract Username that created the VM
        ## Get proper Name from Active Directory
            ## Need access to Active Directory - AD Powershell Module
        ## Update Custom Field / Tag
        ## Save to report
## Update Date Created
    ## Extract Date when VM was created
        ## Convert date to something that can be indexed
        ## Update Custom Field / Tag
        ## Save to report
## Update Owner
    ## Check if Owner was already updated
        ## If not - send email to user that created it to update the field
            ## Need to get email address of creator
            ## SMTP server variable
            ## From address variable
            ## Subject variable
            ## Body to be sent in email
        ## Save notification to report

Create Report

Now to send out the report.

## Take information from report
## Convert to something readable
    ## Export to Excel file
    ## Convert the information to HTML so it can be re-used
## Send Email
    ## SMTP server variable
    ## From address variable
    ## To address variable
    ## Subject variable
    ## Body to be sent in email

Step 4 – Write the code in detail

Now you know what the steps are, it is now time to write the code for each and every step, and this is where most of the work will be. It could be that during the writing of the code you will see that you need to perform additional steps in order to accomplish what you would like, and if so - continue writing out the stages and fill in the appropriate code. Such an example would be - this should be run as a scheduled task which means you will need to provide a method to pass the credentials to the script. Another example would be - you already sent an email to the Creator saying that they should update the Owner, and if they did not - do you leave it? Send them another email? Escalate the issue? For each of these scenarios, there are steps that need to identified, and the appropriate code written.

Another thing I like about this methodology is that it already partially provides some basic documentation for your script - something that is very important - so the code can be re-used in other scenarios as well.

I will not be providing the code for the steps, I will leave that task for you.

Summary

Envision at a high level what you want to accomplish, identify what are the needed stages for this to succeed, and break each of these stages into small steps, and detailed again further into more steps until you have each and every step as part of your plan.

I used a PowerCLI script here as an example - but this methodology can be applied to any project.

• VCDX Certification
• Upgrade from 5.0 to 5.1
• Implementation of vCloud in your organization
• Upgrade Active Directory to 2012

This is the methodology that I use - I hope that it will be of some use to you.

1. Create the .ssh directory under the users /home folder for VM1 and VM2
2. Create an RSA key on VM1 and VM2
3. Copy the contents of ~/.ssh/id_rsa.pub from VM1 and VM2 into ~/.ssh/authorized_keys on both VM1 and VM2
4. You should then be able to connect to each host (and also the localhost as well) without a password prompt.
5. Repeat the process on both VM’s with the oracle user

1. Copying the files between VM1 <-> VM2
2. First connection prompts to add the hosts key to the ~/.ssh/known_hosts file

1. Re-create the ssh_host_rsa_key – the reason for this being – that since these VM’s are deployed from the same template – the ssh_host_rsa_key is identical – and this caused problems for my script (this actually could be useful in some cases – but not here).
2. Create the ~/.ssh/id_rsa.pub key for the root user on each host – without prompts.
3. In order to prevent the popup when connecting to another VM for the first time I needed to get the keys from ssh_host_rsa_key.pub into the .ssh/known_hosts before I connected to the VM for the first time.
4. Add the public key from each VM into the ~/.ssh/authorized_keys file.
5. Get this information from VM1 to VM2 and and vice-versa – and all of this without prompts – which meant I could not go through the guest operating system.
6. Repeat the process for the oracle user.

1. I needed to construct the known_hosts file based on several pieces of information, the hostname, IP address and the ssh_host_rsa_key, I am sure there is a easier way of doing this in bash – but this way works for me.
2. Creating the rsa keys for the oracle user – since I did not want to connect to the VM twice with two different credentials – here I solved the problem by duplicating the files from the root user to the oracle user and manipulated the contents a bit to suit my needs.
3. Copying the files back to the guest after manipulation – resulted in a change in their format from UNIX to DOS and I could not find a way to control that from the PowerCLI side – therefore some vi manipulation was needed to convert them back.

<#
 .SYNOPSIS
  Configure SSH equivalence between two Oracle RAC nodes

 .DESCRIPTION
  The script will execute on both guests, configure the RSA keys,
  known_hosts and authorized_keys files on each host for both the 
  root and oracle user to enable SSH equivalence for Oracle RAC

 .PARAMETER  VM1
  Name of the first VM
 .PARAMETER  VM2
  Name of the first VM
 .PARAMETER  VM1_IP
  The IP address of the first VM
 .PARAMETER  VM2_IP
  The IP address of the second VM
 .PARAMETER  HostCredentials
  The credentials for the ESXi host
 .PARAMETER  GuestCredentials
  The credentials for the guest VM 
 .PARAMETER Cleanup
  Will cleanup the temporary files created. On by default
 
 .EXAMPLE
  PS C:\> Set-SSHKeys -VM1 hosta -VM2 hostb -VM1_IP 10.10.10.1 -VM2_IP 10.10.10.2
  This example shows how to call the Configure-SSHKeys against hosta with the IP address
  of 10.10.10.1 and hostb with the IP address of 10.10.10.2.
 .EXAMPLE
  PS C:\> Set-SSHKeys -VM1 hosta -VM2 hostb -VM1_IP 10.10.10.1 -VM2_IP 10.10.10.2 -HostCredentials `
  (Get-Credential) -GuestCredentials (Get-Credential) -Cleanup:$false
  This example shows how to call the Configure-SSHKeys against hosta with the IP address
  of 10.10.10.1 and hostb with the IP address of 10.10.10.2. while prompting for credentials
  for both the host and the guest and not cleaning up the files after completion.

 .NOTES
  Author: Maish Saidel-Keesing
  Date: 20 January, 2012
  For more in depth info on the script please see:
  http://technodrone.blogspsot.com/2013/01/set-sshkeys.html

#>
function Set-SSHKeys {
 [CmdletBinding()]
 param(
  [Parameter(Position=0, Mandatory=$true)]
  [System.String]$VM1,
  [Parameter(Position=1, Mandatory=$true)]
  [System.String]$VM2,
  [Parameter(Position=2)]
  [System.String]$VM1_IP,
  [Parameter(Position=3)]
  [System.String]$VM2_IP,
  $HostCredentials,
  $GuestCredentials,
  $Cleanup=$true
 )
 # Check for parameters
 if (!$HostCredentials) {
  $HostCredentials = $Host.ui.PromptForCredential("ESXi Host Credentials","Enter the credentials for the ESXi Host","root","")
 }
 if (!$GuestCredentials) {
  $GuestCredentials = $Host.ui.PromptForCredential("Guest VM Credentials","Enter the credentials for the guest VM","root","")
 }
 if (!$VM1_IP) {
 $VM1_IP = (Get-VMGuestNetworkInterface -Name eth0 -vm $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials).IP
 }
 if (!$VM2_IP) {
 $VM2_IP = (Get-VMGuestNetworkInterface -Name eth0 -vm $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials).IP
 }
## script to be executed on VM1
$myscript1 = @"
mv /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_rsa_key.old
mv /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_dsa_key.old
ssh-keygen -t rsa -N "" -f /etc/ssh/ssh_host_rsa_key
ssh-keygen -t dsa -N "" -f /etc/ssh/ssh_host_dsa_key
mkdir ~/.ssh
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
echo -n `$(hostname -s) >> .ssh/known_hosts
echo -n "," >> .ssh/known_hosts
echo -n $VM1_IP >> .ssh/known_hosts
echo -n " " >> .ssh/known_hosts
cat /etc/ssh/ssh_host_rsa_key.pub >> .ssh/known_hosts
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
mkdir /home/oracle/.ssh
cp .ssh/* /home/oracle/.ssh/
chown -R oracle:dba /home/oracle/.ssh
"@
## script to be executed on VM2
$myscript2 = @"
mv /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_rsa_key.old
mv /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_dsa_key.old
ssh-keygen -t rsa -N "" -f /etc/ssh/ssh_host_rsa_key
ssh-keygen -t dsa -N "" -f /etc/ssh/ssh_host_dsa_key
mkdir ~/.ssh
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
echo -n `$(hostname -s) >> .ssh/known_hosts
echo -n "," >> .ssh/known_hosts
echo -n $VM2_IP >> .ssh/known_hosts
echo -n " " >> .ssh/known_hosts
cat /etc/ssh/ssh_host_rsa_key.pub >> .ssh/known_hosts
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
mkdir /home/oracle/.ssh
cp .ssh/* /home/oracle/.ssh/
chown -R oracle:dba /home/oracle/.ssh
"@
 # run the scripts on VM1 and VM2
 Invoke-VMScript -vm $VM1 -ScriptText $myscript1 -ScriptType bash -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Invoke-VMScript -vm $VM2 -ScriptText $myscript2 -ScriptType bash -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 ## authorized_keys for root
 # get files from guests
 Copy-VMGuestFile -GuestToLocal -Source /root/.ssh/authorized_keys -Destination ./authorized_keys_VM1_root -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -GuestToLocal -Source /root/.ssh/authorized_keys -Destination ./authorized_keys_VM2_root -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -GuestToLocal -Source /home/oracle/.ssh/authorized_keys -Destination ./authorized_keys_VM1_ora -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -GuestToLocal -Source /home/oracle/.ssh/authorized_keys -Destination ./authorized_keys_VM2_ora -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 # Change root to oracle to fix running the script with root credentials
 Get-Item .\authorized_keys_*ora | % {
 (get-content $_).Replace("root@","oracle@") | Set-Content $_ -Force
 }
 # concatenate contents of files
 (Get-Content ./authorized_keys_VM1_root) + "`r`n" + (Get-Content ./authorized_keys_VM2_root) + "`r`n" + (Get-Content ./authorized_keys_VM1_ora) + "`r`n" + (Get-Content ./authorized_keys_VM2_ora) | Out-File -FilePath ./authorized_keys -Encoding ascii
 # copy files back 
 Copy-VMGuestFile -LocalToGuest -Source ./authorized_keys -Destination /root/.ssh/ -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials -Force
 Copy-VMGuestFile -LocalToGuest -Source ./authorized_keys -Destination /root/.ssh/ -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials -Force
 $vicmd = "/bin/vi +':w ++ff=unix' +':q' .ssh/authorized_keys"
 $return1 = Invoke-VMScript -ScriptText $vicmd -vm $VM1,$VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 ## known_hosts for root
 # get files from guests
 Copy-VMGuestFile -GuestToLocal -Source /root/.ssh/known_hosts -Destination ./known_hosts_VM1 -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -GuestToLocal -Source /root/.ssh/known_hosts -Destination ./known_hosts_VM2 -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 # concatenate contents of files
 (Get-Content ./known_hosts_VM1) + "`r`n" + (Get-Content ./known_hosts_VM2) | Out-File -FilePath ./known_hosts -Encoding ascii
 # copy files back 
 Copy-VMGuestFile -LocalToGuest -Source ./known_hosts -Destination /root/.ssh/ -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -LocalToGuest -Source ./known_hosts -Destination /root/.ssh/ -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 $vicmd = "/bin/vi +':w ++ff=unix' +':q' .ssh/known_hosts"
 $return1 = Invoke-VMScript -ScriptText $vicmd -vm $VM1,$VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials 
 ## authorized_keys for oracle
 # copy files back 
 Copy-VMGuestFile -LocalToGuest -Source ./authorized_keys -Destination /home/oracle/.ssh/ -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials -Force
 Copy-VMGuestFile -LocalToGuest -Source ./authorized_keys -Destination /home/oracle/.ssh/ -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials -Force
 $vicmd = "/bin/vi +':w ++ff=unix' +':q' /home/oracle/.ssh/authorized_keys"
 $return1 = Invoke-VMScript -ScriptText $vicmd -vm $VM1,$VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials 
 ## known_hosts for Oracle
 # copy files back 
 Copy-VMGuestFile -LocalToGuest -Source ./known_hosts -Destination /home/oracle/.ssh/ -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -LocalToGuest -Source ./known_hosts -Destination /home/oracle/.ssh/ -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 $vicmd = "/bin/vi +':w ++ff=unix' +':q' /home/oracle/.ssh/known_hosts"
 $return1 = Invoke-VMScript -ScriptText $vicmd -vm $VM1,$VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials 
 # remove temporary files
 if ($Cleanup) {
  Get-Item .\authorized_keys*, .\known_hosts* | Remove-Item -Confirm:$false 
 }
}

A bit more details about the script that is run on the guest. 

Lines 73-76 - The guest SSH keys are re-created as I explained above. 

Lines 77-78 – Create the .ssh directory and create the keys. –N is to set a blank password on the key and –f is for the path. 

Lines 79-83 - The known_hosts file is basically a concatenation of 3 things for each entry:
<hostname>, <IP_Address> <Contents of rsa_key.pub> (The commas and spaces are important!) 

Line 84 - Add the contents of id_rsa.pub to the authorized_keys file. 

Lines 85-87 – Copy the files into the oracle user’s directory and make sure sure the file ownership is correct.

I brought this up on Twitter a while ago.

What network path will the transfer take when using Copy-DatastoreItem - does anyone know?

— Maish Saidel-Keesing (@maishsk) January 8, 2013

I studied the traffic flow – and would like to share it with you here, but first here is the architecture of the testbed – which will help explain in more detail

Well the environment is pretty simple. One vCenter server, four ESXi Servers – each with a local datastore and a shared datastore among the 4 hosts. My laptop is the one that was running the Copy-DatastoreItem cmdlet.

Copy-DatastoreItem has the following parameters

• Item - Specify the datastore item you want to copy. You can use a string to provide a relative path to the item in the current provider location.
• Destination - Specify the destination where you want to copy the datastore item.

The PowerCLI syntax:

Copy-DatastoreItem -Item $ISOPath -Destination $Destination

The first case was copying to a local datastore on ESX3 (x.x.x.173)

So I fired off the command and the ISO started to copy.

But how does the file get from my laptop to the ESX3? From my laptop directly to the host? Some other way?

In order to check this I chose a file that was reasonably large (~3.0 GB) so I could see the network activity. I opened up the Resource Monitor and sorted the columns to see the network traffic and saw the following.

The network traffic was going from my laptop directly to the vCenter Server – using Powershell of course.

Next – from the vCenter – is had to get to the host of course.. – but how?

Here is the Resource monitor from the vCenter Server

On the top right what you see is that the vpxd.exe (vCenter server process) is the one that has the highest amount of network traffic.

In the bottom window you can see on the first line that it is sending out traffic to ESX3 (.173) over port 902 and on the second line it is receiving traffic from my laptop (.187) over port 443. This makes perfect sense.

PowerCLI is communicating with the vCenter – and the vCenter is sending the traffic over to the host. So if we were to look at the original architecture again , the traffic flow will look like this

Next I tried the same thing, but this time to the Shared datastore.

Based on what we had before the flow should be:

Laptop –> vCenter –> ESX host.

But the question is though – which ESXi host? The destination parameter to where I want to copy the file is as follows:

vmstore:/<datacenter_name>/<datastore_name> –
in my case – vmstore:/UCS/VMGuestDatastore

There is no indication to which host this will go to, there are a number of hosts in the cluster and they are all connected to same datastore. So how does the traffic actually flow in the end?

To answer that we need to look at the Resource Monitor on the vCenter again.

As you will notice – the same traffic flow – but in this case – it changed to ESX2 (.172) and was not going through ESX3 anymore.

I tried the process a number of times – and the result was always the same, it always went to ESX2 (.172)

I could not find the logic behind this.

It did not take the hosts sorted by Name nor according to HostId as you can see below.

This one has me intrigued – and if anyone has any ideas – what the logic is behind how vCenter chooses which host the traffic will flow through – please do share.

Just a heads up. According to the Release Notes PowerCLI does is not supported

Does this mean that it will not work – No of course not!  From what I have tested it works almost flawlessly – but there are some quirks…

For example - Set-NetworkAdapter returning 'Operation is not valid due to the current state of the object' and Error with Move-VM: Operation is not valid due to the current state of the object.

And as Luc put it, “Well, you can't file a bug for something that is not supported, now can you “

Update – February 12th, 2013

VMware have now released an updated version – see the announcement here - PowerCLI 5.1 Release 2 Now Available

This will actually render this blog post obsolete – but I am happy VMware have addressed this issue.

Luc Dekens wrote a great post a while back Will Invoke-VMScript work? about the prerequisites needed in order to get Invoke-VMscript to work. Stop for a minute and go and read his post.

Glad to have back.

As part of an Oracle RAC provisioning script that I am working on – one of the first things I wanted to do was to configure the network settings for my two nodes – with parameters taken from a config file.

Of course if the VM does not have an IP address yet then you cannot configure it through the network, so here is where Invoke-VMscript comes into play.

A few things first off the bat. My configuration was working also with the 32-bit engine but also with the 64-bit engine as well. The rest of the prerequisites were all there.

So here is what was happening. In the script I had stored the HostCredentials and the Guestcredentials each in a variable. When it came time power on the VM’s,The script would wait for the VMware tools to start running in the guest before executing the command and then run my script inside the guest OS – but the command would fail with this message.

Invoke-VMScript : 02/01/2013 15:17:07    Invoke-VMScript        Error occured while executing script on guest OS in VM
'testdbCA1b'. Could not locate "Powershell" script interpreter in any of the expected locations. Probably you do not have enough permissions to execute command within guest.
At line:5 char:1
+ Invoke-VMScript -ScriptText $bb -vm $dbvm2 -HostCredential $hostcreds -GuestCred ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (testdbCA1b:VirtualMachineImpl) [Invoke-VMScript], VimException
    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_RunScriptCore_ExeLookupFailed,VMware.VimAutomation.ViCore.Cmdlets.Commands.InvokeVmScript

Now this was really weird – because this was completely not true. To make it even more baffling – when trying to run the commands manually – not as part of the script – it would work – without a problem. So I was starting to think that perhaps there was a problem that the credentials were not being passed properly down to the command – which was not likely – but still I had no other clue.

I then wondered – Invoke-VMscript – interacts with the VMware Tools in the guest through VIX – so maybe there was a problem there.

So I checked my versions (it was 1.12) and looked at the release notes and saw something there that ultimately led me onto the right path.

OK I was not getting any of these errors, but my command would not work. So I wanted to see what the logs of the VMware Tools in the guest were saying – after all it was interacting with the guest through VMware Tools. But where was the log?

KB1007873 - Enabling debug logging for VMware Tools within a guest operating system showed me the way to enable VMware Tools logging in a Linux VM.

You need to create a file (if it does not exist) /etc/vmware-tools/tools.conf and add to that file:

log = true
log.file = /tmp/vmtools.log

I then performed the following to test my theory.

1. Restart the VM
2. Wait for the tools to report Running and up-to-date
3. Invoke-VMscript

This I did with a simple PowerShell script.

Restart-VM -VM $dbvm2 -Confirm :$false
Sleep 10
while (((get-vm $dbvm2 ).ExtensionData.Guest.ToolsRunningStatus ) -ne "guestToolsRunning" ) {
Write-Host "....." -ForegroundColor Yellow
Sleep 5
}

Invoke-VMScript -ScriptText $bb -vm $dbvm2 -HostCredential $hostcreds -GuestCredential $guestcreds
while ( $? -eq $false ) {
Get-Date -Format HH :mm :ss
sleep 2
Invoke-VMScript -ScriptText $bb -vm $dbvm2 -HostCredential $hostcreds -GuestCredential $guestcreds
}

So I noticed a few things.

1. VMware Tools comes up and reports itself as running – way before the OS is actually available – and that you have a console prompt.
   Even before SSH starts
2. The first 3-5 tries of Invoke-VMscript would fail – with the same error message I had before. And suddenly it would work as if nothing was wrong.

I went to look in the VMware Tools log that I had just configured and there I found something which I find very strange – but did solve my mystery but I still do not have the answer as to why it is happening.

For the failed attempts I had this in the log.

[Jan 02 13:17:06.925] [   debug] [vix] VixTools_StartProgram: args: progamPath: 'cmd.exe', arguments: '/C powershell -NonInteractive -EncodedCommand cABvAHcAZQByAHMAaABlAGwAbAAuAGUAeABlACAALQBPAHUAdABwAHUAdABGAG8AcgBtAGEAdAAgAHQAZQB4AHQAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0AQwBvAG0AbQBhAG4AZAAgACcAJgAgAHsAbABzACAALQBsAGEAfQAnACAAPgAgACIALwB0AG0AcAAvAHAAbwB3AGUAcgBjAGwAaQB2AG0AdwBhAHIAZQAwACIAOwAgAGUAeABpAHQAIAAkAGwAYQBzAHQAZQB4AGkAdABjAG8AZABlAA=='', workingDir: '

[Jan 02 13:17:11.988] [   debug] [vix] VixTools_StartProgram: args: progamPath: 'cmd.exe', arguments: '/C powershell -NonInteractive -EncodedCommand cABvAHcAZQByAHMAaABlAGwAbAAuAGUAeABlACAALQBPAHUAdABwAHUAdABGAG8AcgBtAGEAdAAgAHQAZQB4AHQAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0AQwBvAG0AbQBhAG4AZAAgACcAJgAgAHsAbABzACAALQBsAGEAfQAnACAAPgAgACIALwB0AG0AcAAvAHAAbwB3AGUAcgBjAGwAaQB2AG0AdwBhAHIAZQAwACIAOwAgAGUAeABpAHQAIAAkAGwAYQBzAHQAZQB4AGkAdABjAG8AZABlAA=='', workingDir: '

But for the successful attempt the log showed (which was what I expected)

[Jan 02 13:17:23.211] [   debug] [vix] VixTools_StartProgram: args: progamPath: '/bin/bash', arguments: '-c "bash > /tmp/powerclivmware0 2>&1 -c \"ls -la\""'', workingDir: '
[Jan 02 13:17:23.211] [   debug] [vmsvc] Executing async command: '"/bin/bash" -c "bash > /tmp/powerclivmware0 2>&1 -c \"ls -la\""' in working dir '/root'
[Jan 02 13:17:23.214] [   debug] [vix] VixToolsStartProgramImpl started '"/bin/bash" -c "bash > /tmp/powerclivmware0 2>&1 -c \"ls -la\""', pid 3792

Now here is the weird part. If you look at the first two failures – you will see that VIX trying to execute a Windows command on a Linux operating system – which…. probably .. won’t…. really…. work…. !!!

Only about 20 seconds later – did it execute the correct bash command – in my case ‘ls –la’ and it worked of course.

So here I found my way around my problem – but have not gotten to the bottom of the mystery yet. I put in an extra sleep statement into the script that would wait a bit longer until the OS was completely up and only then run the Invoke-VMscript command – and all was working fine…

So a few things I learned today:

• How to enable logging for VMware Tools
• VIX does weird things.
• A workaround is as good of a solution as any other.
• I would add one more thing to Luc’s prerequisites – wait until the VM has completely started before attempting to use Invoke-VMscript.

Hey…. -  that is not possible – I hear you say – well in principle you are right. Up until today…

By mistake of course – I found that there was a change made to the 5.1 release of PowerCLI – but this change has not been documented anywhere – which I think is a shame. This post is the only public reference I know of.

Up until the 5.1 release you could not create an EagerZeroedThick hard disk with PowerCLI. Let’s look at the 5.0 documentation for PowerCLI.

As you can see above the options are Thin or Thick and if you go to look at the
VirtualDiskStorageFormat – Enum you will see the that there is no EagerZeroedThick option.

The PowerCLI changelog has a new Cmdlet Move-Harddisk – which as you can see allows you to migrate a VMDK from one location to another – and if you also noticed..

Yep, EagerZeroedThick is one of the options as you can see above. The VirtualDiskStorageFormat – Enum was not updated though.

So I used this today as part of a bigger automation process to prepare some VM’s for Oracle RAC (which I will post about in the not too distant future)

get-vm $vm1 | New-HardDisk -DiskType flat -CapacityGB 2 -StorageFormat EagerZeroedThick -Datastore $dbds |New-ScsiController -Type ParaVirtual -BusSharingMode NoSharing

Which is so much easier than… (taken from Luc Deken’s post)

$vmName = <vm-name>
$vCenter = <vCenter-name>
$esxAccount = <ESX-account>
$esxPasswd = <ESX-password>

function Set-EagerZeroThick{
	param($vcName, $vmName, $hdName)

# Find ESX host for VM
	$vcHost = Connect-VIServer -Server $vcName -Credential (Get-Credential -Credential "vCenter account")
	$vmImpl = Get-VM $vmName
	if($vmImpl.PowerState -ne "PoweredOff"){
		Write-Host "Guest must be powered off to use this script !" -ForegroundColor red
		return $false
	}

	$vm = $vmImpl | Get-View
	$esxName = (Get-View $vm.Runtime.Host).Name
# Find datastore path
	$dev = $vm.Config.Hardware.Device | where {$_.DeviceInfo.Label -eq $hdName}
	if($dev.Backing.thinProvisioned){
		return $false
	}
	$hdPath = $dev.Backing.FileName

# For Virtual Disk Manager we need to connect to the ESX server
	$esxHost = Connect-VIServer -Server $esxName -User $esxAccount -Password $esxPasswd

# Convert HD
	$vDiskMgr = Get-View -Id (Get-View ServiceInstance -Server $esxHost).Content.VirtualDiskManager
	$dc = Get-Datacenter -Server $esxHost | Get-View
	$taskMoRef = $vDiskMgr.EagerZeroVirtualDisk_Task($hdPath, $dc.MoRef)
	$task = Get-View $taskMoRef
	while("running","queued" -contains $task.Info.State){
		$task.UpdateViewData("Info")
	}

	Disconnect-VIServer -Server $esxHost -Confirm:$false

# Connect to the vCenter
	Connect-VIServer -Server $vcName -Credential (Get-Credential -Credential "vCenter account")
	if($task.Info.State -eq "success"){
		return $true
	}
	else{
		return $false
	}
}

Set-EagerZeroThick $vCenter $vmName "Hard disk 1"

Ah the little things…..

I wonder what other gems are hidden deep in the bowels of PowerCLI that were not updated in the changelog/release notes?? …. What do you think ??

I am currently preparing a deployment package architecture for a full vSphere environment and one of the requests was to include PowerCLI in the installation script.

I was surprised that I could not find anything already mentioned on how to do this on Google.

So here is the syntax:

# Install PowerCLI
Set-executionPolicy RemoteSigned -Confirm:$false -force
Write-Host "Installing PowerCLI"
$myargs = $myargs = '/q /s /w /L1033 /v" /qn '
$exe = "C:\installs\VMware-PowerCLI-5.1.0-793510.exe"
Start-process $exe $myargs –Wait
Add-PSSnapin -Name VMware.VimAutomation.Core

Line 4-5: Prepare the installation syntax

Line 7: Add the PowerCLI Snapin

Easy as that!

I actually do not understand why I have not put this in a blog post before, but it is about time - because I used it again today.

Sometime you need to store a credential for a number of purposes, be it a scheduled script - or just not having to enter credentials each and every time you would like to connect to a Host or your vCenter.

PowerCLI has it's own credential store with the New-VICredentialStoreItem and Get-VICredentialStoreItem cmdlets.

Personally - I do not like using this Cmdlet and have another method that I prefer using -System.Management.Automation.PSCredential.

First get the credential and store it in a file.

(Get-Credential).Password | ConvertFrom-SecureString | Out-File -FilePath C:\users\msaidelk\Documents\scripts\maish.cred

Get-Content C:\users\msaidelk\Documents\scripts\maish.cred 01000000d08c9ddf0115d1118c7a00c04fc297eb01000000b215c3e1ee044b4286513bf7017abc8f0000000002000000000003660000c000000010000000d6533af1a8d6da153c0bf43713400cb30000000004800000a0000000100000008217a9ea26e47eba311ce8272156a7a2180000004c7047e3ca25bc540078e9dad60b0aff8ba48f37709534a614000000e582d7d6522c111d7101849 a27f8a9c034eb4ab6

$vicred = New-Object System.Management.Automation.PsCredential "root", (Get-Content "C:\Users\msaidelk\Documents\scripts\maish.cred" | ConvertTo-SecureString) 

PS C:\Users\testa> $vicred = New-Object System.Management.Automation.PsCredential "root", (Get-Content "C:\temp\maish.cred" | ConvertTo-SecureString) ConvertTo-SecureString : Key not valid for use in specified state. At line:1 char:108 + ... \maish.cred" | ConvertTo-SecureString) + ~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (:) [ConvertTo-SecureString], C ryptographicException + FullyQualifiedErrorId : ImportSecureString_InvalidArgument_Cryptographic Error,Microsoft.PowerShell.Commands.ConvertToSecureStringCommand

You could do use this method Storing Passwords to Disk in PowerShell with Machine-key Encryption but that is overkill for my use case.

To summarize the process:

1. Store the password in a file.
2. Build the credential by providing the correct username and the content of the file and store it in a variable.
3. Use the variable when connecting to your vCenter / Hosts.

In my Powershell $PROFILE I have the following three lines:

$vcenter = "msaidelk-lab1" $vicred = New-Object System.Management.Automation.PsCredential "MAISHSK\Maish", (Get-Content "C:\Users\msaidelk\Documents\scripts\maish.cred" | ConvertTo-SecureString) Connect-VIServer $vcenter -Credential $vicred