2009-11-30

Deep Security 7 - Trend Micro & VMsafe

Trend Micro have now joined Altor Networks and Reflex Systems with their new offering that utilizes VMware's VMsafe technology. I expect we will be seeing more and more of the Security Companies release their products that will utilize VMsafe

From the Product Page

Protect physical, virtual and cloud servers from malicious attack

 

Trend Micro Deep Security 7 provides advanced protection for servers in the dynamic datacenter, whether physical, virtual or in the cloud. Brought to Trend Micro through the acquisition of Third Brigade, Deep Security combines intrusion detection and prevention, firewall, integrity monitoring and log inspection capabilities in a single, centrally managed software agent.

Deep Security protects confidential data and critical applications to help prevent data breaches and ensure business continuity, while enabling compliance with important standards and regulations such as PCI, FISMA and HIPAA. Whether implemented as software, virtual appliance, or in a hybrid approach, this solution equips enterprises to identify suspicious activity and behavior, and take proactive or preventive measures to ensure the security of the datacenter.

From the datasheet

ARCHITECTURE

  • Deep Security Virtual Appliance. Transparently enforces security policies on VMware vSphere
    virtual machines for IDS/IPS, web application protection, application control, and firewall protection—
    coordinating with Deep Security Agent, if desired, for integrity monitoring and log inspection.
  • Deep Security Agent. This small software component deployed on the server or virtual machine
    being protected enforces the datacenter’s security policy (IDS/IPS, web application protection,
    application control, firewall, integrity monitoring, and log inspection).
  • Deep Security Manager. Powerful, centralized management enables administrators to create
    security profiles and apply them to servers, monitor alerts and preventive actions taken in
    response to threats, distribute security updates to servers, and generate reports. New Event
    Tagging functionality streamlines the management of high-volume events.

DEPLOYMENT AND INTEGRATION

  • VMware integration with VMware vCenter and ESX Server enables organizational and operational
    information to be imported into Deep Security Manager, and detailed security to be applied to an
    enterprise’s VMware infrastructure
  • Integration with VMsafe™ APIs enables rapid deployment on ESX servers as a virtual appliance to
    immediately and transparently protect vSphere virtual machines
  • Detailed, server-level security events are provided to a SIEM system, including ArcSight™, Intellitactics, NetIQ, RSA Envision, Q1Labs, Loglogic, and other systems through multiple integration options
  • Directory integration with enterprise-scale directories, including Microsoft Active Directory
  • Configurable management communication minimizes or eliminates firewall changes typically
    needed for centrally managed systems by enabling either the Manager or the Agent to initiate
    communication
  • Agent software can be deployed easily through standard software distribution mechanisms such
    as Microsoft® SMS, Novel Zenworks, and Altiris.

DEEP SECURITY MODULES

Deep Packet Inspection

  • Examines all incoming and outgoing traffic for protocol deviations, content that signals an attack, or policy violations
  • Operates in detection or prevention mode to protect operating systems and enterprise application vulnerabilities
  • Defends against application-layer attacks, SQL injection, and cross-site scripting
  • Provides valuable information, including who attacked, when they attacked, and what they attempted to exploit
  • Automatically notifies administrators when an incident has occurred Intrusion Detection and Prevention
  • Protects against known and zero-day attacks by shielding known vulnerabilities from unlimited exploits
  • Automatically shields newly discovered vulnerabilities within hours, pushing protection to thousands of servers in minutes without a system reboot
  • Includes out-of-the-box vulnerability protection for over 100 applications, including database, web, email, and FTP servers
  • Smart rules provide zero-day protection from unknown exploits that attack an unknown vulnerability, by detecting unusual protocol data containing malicious code Integrity Monitoring
  • Monitors critical operating system and application files, such as directories, registry keys, and values, to detect malicious and unexpected changes
  • Detects modifications to existing file systems and new file creations and reports them in real time
  • Enables on-demand, scheduled or realtime detection, checks file properties (PCI 10.5.5), and monitors specific directories
  • Delivers flexible and practical monitoring through includes/excludes and auditable reports Web Application Protection
  • Assists compliance (PCI DSS 6.6) to protect web applications and the data they process
    Defends against SQL injection, cross-site scripting, and other web application vulnerabilities
  • Shields against vulnerabilities until code fixes can be completed

Application Control

  • Provides increased visibility into, or control over applications accessing the network
  • Uses application control rules to identify malicious software accessing the network
  • Reduces vulnerability exposure of servers

Bidirectional Stateful Firewall

  • Decreases the attack surface of physical, cloud, and virtual servers
  • Centrally manages server firewall policy, including templates for common server types
  • Features fine-grained filtering (IP and MAC addresses, ports), design policies per network interface, and location awareness
  • Prevents denial of service attacks and detects reconnaissance scans
  • Covers all IP-based protocols (TCP, UDP, ICMP, etc.) and all frame types (IP, ARP, etc.)

Log Inspection

  • Collects and analyzes operating system and application logs for security events
  • Assists compliance (PCI DSS 10.6) to optimize the identification of important security events buried in multiple log entries
  • Forwards events to SIEM system or centralized logging server for correlation, reporting, and archiving
  • Detects suspicious behavior, collects security events and administrative actions across your datacenter, and creates advanced rules using OSSEC syntax

2009-11-26

Testing Network Failover with PowerCLI

Two weeks ago I wrote an article about How To Bring Down A Single NIC In ESX?. In that post you could see that in order to test this you had to go into the console of the ESX and run the commands on the console.

Already then I was thinking, why not do this from PowerCLI, without having to log into each host.

So here we go..

For the examples sake:

VI Server vcenter.maishsk.local
ESX Host for Testing esx1.maishsk.local
vmnic for Redundancy test vmnic2

 

   1: function Test-NetworkFO ($vmhost, $vmnic, $switch) {
   2:     
   3:     If ($switch){
   4:     ##Define which NIC should be tested   
   5:     $mynic = Get-VMHostNetworkAdapter -VMHost (get-vmhost $vmhost) | `
   6:     Where-Object { $_.DeviceName -eq $vmnic }   
   7:  
   8:     ##Bring the NIC down   
   9:     Set-VMHostNetworkAdapter -PhysicalNic $mynic -Duplex Half -BitRatePerSecMb 10  
  10:     }
  11:     
  12:     If (-not $switch){
  13:     ##Define which NIC should be tested   
  14:     $mynic = Get-VMHostNetworkAdapter -VMHost (get-vmhost $vmhost) | `
  15:     Where-Object { $_.DeviceName -eq $vmnic }   
  16:  
  17:     ##Bring the NIC back up  
  18:     Set-VMHostNetworkAdapter -PhysicalNic $mynic -AutoNegotiate
  19:     }
  20: }
  21:  
  22: # Turn it on
  23: Test-NetworkF0 "vcenter.maishsk.local" "vmnic2" $true
  24:  
  25: # Turn it off
  26: Test-NetworkF0 "vcenter.maishsk.local" "vmnic2" $false


This is a way to test your Network Redundancy for your ESX hosts.

I would like to thank Alan Renouf - who helped me clean up the code and put it into a function.

Have a happy Thanksgiving and a good weekend!

My Journey to vSphere - The Beginning

My policy is not to update to new releases until they have stabilized for a decent period of time.

Therefore, we have not yet performed our upgrade to Version 4.0 yet. We were waiting for the release of Update 1 before migrating the system to version 4.

I have been using ESX4 since the private Beta, but not on a production environment.

Over the upcoming period I will share with you the process of the upgrade from 3.5 to 4.1.

First off, I will not perform the upgrade until the latest release stabilizes a bit, and the "oversights" that have and will pop up - calm down. I actually find this a good sound idea,

But just to lay out the Migration Plan:

  1. Installation of a new vCenter - 4.1
  2. Migration of environment to the new vCenter (this will probably be the most lengthy part)
  3. Upgrade ESX Hosts 1 by 1
  4. Upgrade VM's to Virtual Hardware Version 7
  5. Project complete.

It may look simple - but when dealing with a production environment and with so many dependencies - nothing is that simple.

I hope that the sharing of my experiences will be able to help those in their journey through the virtual universe (and also as a by-product, document my stages for the future reference).

2009-11-25

"Need have have" - revisited

If you remember about 6 months ago I wrote an article about the fact that hot add of any kind of virtual hardware in the free free version of ESXi. I thought I would revisit the problems I had now that Update 1 was released last week.

The host has been upgraded to ESX4i Update 1

image

I took a Windows 2008 Guest OS.

image

Firstly, I noticed - CPU Hot plug was not available. Regarding memory Hot Add - I did have the option to turn on / off.

I powered on the machine. And started my journey again to add a new hard disk to the VM while it was powered on.

image

image

Edit Settings -> Add -> yada yada yada….. (you know the drill…)

I did not receive any errors

image

And Lo and Behold:

image

Tried the same with a second NIC

image

And Lo and Behold:

image 

Last thing I tried - (you remember the memory/CPU hot-add from before.

This is not a feature that is offered with the free version of ESXi - but what the heck - the option was there, so I figured I would try it.

image
image

And I got shot down - as expected:

image

The questions I asked my previous article were:

  1. The error message of course is not very well phrased - and I would love to know what "need have have" is?
  2. Why in heaven's name would you take away the most basic functionality that has been in every single VMware product (besides Player) for the past 5 years? VMware? Please tell me that was a mistake?

So as you can see, the first has been fixed, and William Shakespeare will no longer be turning over in his grave.

Regarding the second one, I guess VMware realized the mistake, and have remedied the situation that arose with ESX4i. Thank you for correcting the problem

And of course - the removal of the above mentioned hardware (HD and NIC) works on a live machine as well.

Hope you enjoyed the ride!

2009-11-23

71-663 - Exchange 2010 Beta exam

71-663: Pro: Designing and Deploying Messaging Solutions with Microsoft Exchange Server 2010. If you pass the beta exam, the exam credit will be added to your transcript and you will not need to take the exam in its released form.

Availability

Registration begins: November 23, 2009

Beta exam period runs: December 1, 2009– December 21, 2009

Receiving this invitation does not guarantee you a seat in the beta; we recommend that you register immediately. Beta exams have limited availability and are operated under a first-come-first-served basis. Once all beta slots are filled, no additional seats will be offered.

Testing is held at Prometric testing centers worldwide, although this exam may not be available in all countries (see Regional Restrictions). All testing centers will have the capability to offer this exam in its live version.

Registration Information

You must register at least 24 hours prior to taking the exam.
Please use the following promotional code when registering for the exam: EPROJ
Receiving this invitation does not guarantee you a seat in the beta; we recommend that you register immediately.

Originally posted here

I have already signed up for this exam - it helps to get a personal invitation :)

Disconnected ESX Host

Got a call today.

Panic!!!!

All VM on an ESX host just went grey – all disconnected.

Trouble shooting steps:

  1. Ping ESX host Service Console – All ok
  2. Look in the VI client what is with the server – NOT OK – all machines are greyed out – (hey that is what they said wasn’t it).
  3. SSH into the Service console - All ok
  4. Direct GUI management to the server NOT OK. could not load the inventory
  5. All VM's on the host were running and responding to ping.
  6. No failover was initiated in the cluster.
  7. On the console – I saw that there were 7 processes of vmware-hostd each using a lot of RAM.
  8. service mgmt-vmware stop – to stop the service. GOT STUCK
  9. Off to this KB  which helped me stop the service and get the host responsive again.

    # cd /var/run/vmware
    # ls -l vmware-hostd.PID watchdog-hostd.PID (to get the current PID of the process)
    # cat vmware-hostd.PID (i.e. 1234 is the PID)
    # kill -9 <PID> (kill the process)
    # rm vmware-hostd.PID watchdog-hostd.PID remove the files
    # service mgmt-vmware start (restart the agent)
  10. The host came back online – all VM’s were no longer grey.

Here starts my questions.

  1. Why did this happen?

    I went to start digging into the logs and found that there was a gap in the system logs for about 20 minutes – which is really strange.
  2. It seems this happened after a snapshot removal

    image 
  3. I have opened a SR with VMware to get to the bottom of this issue.

2009-11-19

The release that was (NOT!)

Ok so the detective work was not 100% correct. If it was not for unforeseen circumstances then we should have had a release today

So why was there no release?

The only thing I found was from Andre Leibovici’s Blog

VMware uncovered some upgrade issues from View 3 to View 4 and the new unofficial date slipped to Monday 23/11. It also looks like the vSphere 4.0 Update 1 has also slipped to the same date.

Personally I would prefer the wait instead of having to deal with issues because of the new build.

It is ok, only a few more days to go.

Is today the day for Update 1??

No official sources, but a small bit of detective work.

I hate rumors, but it has already started on Twitter.

Trickles of info also from here  and here.

The talk is that View 4.0 will only work with ESX(i) 4 Update 1. View will be released tomorrow.

image

From the View Press Release

Pricing and Availability

VMware View 4 will be generally available on November 19, 2009 through VMware sales and the company’s extensive network of OEM partners and channel partners.

So officially it is November 19th, 2009 at least in my part of the globe - so get ready….

Put 2 + 2 together and what do you get? Vmware View 4.0 and ESX 4 Update 1!

A whole lot of Virtualization happiness.

2009-11-17

Access Console in Linux Guest

Have you ever had the issue of wanting to go into the console of a Linux Virtual Machine that is running at RUNLEVEL 5 (GUI) - because you are one of those Linux geeks who think that GUI is for the weak?

Seriously though - sometimes you need to access the console - to try and fix up something that cannot be done in the GUI.

On a Linux host you would usually press CTRL+ALT+F1 (of Fx - depending on which console you would like). It doesn't work does it?

Hold Ctrl+Alt and hit Space and then F1 (or Fx) without releasing Ctrl+Alt.
This will invoke the Ctrl+Alt+F1 on the guest OS.

2009-11-16

ONYX - Go forth and PowerCLI'ze!

I have been waiting for this ever since it was announced at VMWorld 2009.

Carter Shanklin just announced tonight!

Been waiting to try "Project Onyx" the PowerCLI script generator for VMware vSphere? Check out http://is.gd/4Wxmc

Here is a small Demo

Go forth and PowerCLI'ze !!

Yep I passed! - My VCDX Admin Exam

On October 26th, I sat the VCDX Administration Exam. Today I received notification that I passed.

Now if you were following me on Twitter then you would have seen that I was not at all happy with the feeling I left the Exam with.

My experience.

It was taken in the Pearson testing Center in Tel-Aviv. This was the first time I have taken an exam there. Firstly, I am grateful that I could do the Exam in Israel, and did not have to fly to another country to actually sit the Exam.

I have taken exams before, but for the first time, I think, this is the only place that really abided 100% (TO THE LETTER!!!!!) to all the rules (Well done!). I felt like I was in a fish tank. Constantly being watched and with two surveillance cameras on me from the roof at all times. Adding to the fact that this was an important exam, feeling relaxed was not part of the experience.

Of course I will not be able to provide you any of the questions that were on the Exam, that would be violating the NDA, but I will add to the common knowledge already out there.

The exam time limit was 2 hours and 45 minutes. 70 Multiple choice questions and 11 Hands-on lab. I distinctly remember that the split was 50-50 (questions-labs) and not 60-40 as I have seen on several places on the www.

I as well finished the Exam with not more than 60 seconds to spare. You have to manage and divide your own time between the two parts.

I will say that there were questions / Labs on almost every part of the blueprint, some more than others.

You will need to know your command syntax and remember that there is more than one way to do things.

The VCDX is a 4 part process (http://mylearn.vmware.com/portals/certification/)

  1. You must be certified as a VMware Certified Professional (VCP) on VMware Infrastructure 3 and you must complete the Qualification Skills Review.
  2. Pass the VMware Enterprise Administration Exam. Includes live labs and tests a higher level of skill set than the exam for VCP on VMware Infrastructure 3.
  3. Pass the VMware Design Exam. This exam focuses on actual design scenarios and contains simulations and situational questions.
  4. Submit, present and defend a successful VMware Infrastructure design and implementation plan.

The resources I used for studying for this Exam:

  1. Previous knowledge and experience using & Designing vSphere and ESX infrastructures.
  2. VMware Enterprise Administration Exam Blueprint, v3.5
  3. Some great revision guides from
  4. 1 HS21 Blade with 8GB RAM + 2x73gb disks installed with ESX4i
  5. 8-10 VM's
    • 2x ESX 3.5
    • 1x Windows 2003 - Domain Controller
    • 1x Windows 2003 - vCenter Server
    • 1x Openfiler - Shared Storage
    • 1x Windows 2003 - VCB server
    • 1-2 VM's in each virtual ESX Server
    • "….And a partridge in a pear tree…"

On to the Design Exam. Oh and I would like to squeeze in the VCP4 in the next 6 weeks as well :)

2009-11-12

How To Bring Down A Single NIC In ESX?

I was installing a new ESX Host today, and before putting it into Production, I wanted to test that the Network Failover was working correctly.

Now instead of going over to the server and pulling out network cables, I started to think, there must be a way to do this (like all good administrators do) from the comfort of my desk.

Easiest would be – Log into the switch and shutdown the port. I agree, but what happens when your the Virtualization Team do not control the Physical network (my situation)? That was not an option.

I started to look if I could bring the interface down.

With regular Linux you would – ifdown eth0

[root@esx1 ~]# ifdown eth0
usage: ifdown <device name>

So that did not work! Pretty obvious why though… There is no eth0 interface.

[root@esx1 ~]# ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:784233 errors:0 dropped:0 overruns:0 frame:0
          TX packets:784233 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3017478364 (2877.6 Mb)  ...:3017478364 (2877.6 Mb)

vmnic0    Link encap:Ethernet  HWaddr 00:1A:64:E5:31:38
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:6602129 errors:0 dropped:0 overruns:0 frame:0
          TX packets:237757 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1130250120 (1077.8 Mb) ..es:195027359 (185.9 Mb)
          Interrupt:113 Memory:92000000-92012100

vmnic1    Link encap:Ethernet  HWaddr 00:1A:64:E5:31:3A
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:505 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:303685 (296.5 Kb)  TX bytes:0 (0.0 b)
          Interrupt:201 Memory:94000000-94012100

vswif0    Link encap:Ethernet  HWaddr 00:50:56:4C:43:11
          inet addr:1.1.1.1  Bcast:1.1.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2490053 errors:0 dropped:0 overruns:0 frame:0
          TX packets:260404 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:664363188 (633.5 Mb)  TX bytes:196222902 (187.1 Mb)

 

So I tried..

[root@esx1 ~]# ifdown vmnic0
usage: ifdown <device name>

And that did not work either.

I did not want to shutdown the whole vSwitch – because then I would lose the connectivity (of course there is the possibility of creating the second Service Console – but this was not the case here)

The only way to do this is to manipulate the NIC’s directly. So what can you do with the physical NIC’s?

[root@esx1 ~]# esxcfg-nics
esxcfg-nics <options> [nic]
   -s|--speed <speed>      Set the speed of this NIC to one of 10/100/1000/10000.
                           Requires a NIC parameter.
   -d|--duplex <duplex>    Set the duplex of this NIC to one of 'full' or 'half'.
                           Requires a NIC parameter.
   -a|--auto               Set speed and duplexity automatically.  Requires a NIC parameter.
   -l|--list               Print the list of NICs and their settings.
   -r|--restore            Restore the nics configured speed/duplex settings (INTERNAL ONLY)
   -h|--help               Display this message.

 

So you cannot shut down the device, but you do the next best thing – set it to 10 Half-Duplex.

That will bring the port down.

[root@esx1 ~]#esxcfg-nics -s 10 -d half vmnic2



image

And to bring it back up

[root@esx1 ~]# esxcfg-nics -a vmnic2

Now unless there is a good reason – your ports should always be set to Auto-Negotiate, and in this casem when the nic was reset to auto – then it came back up without any issues.

The failover worked – so my experiment was successful.

image

Some might call it being a lazy Bum – I call it being a Good Administrator!!

Hope you enjoyed the ride!

2009-11-11

vSphere Client for Windows 7 and Server 2008 R2

VMware have released an official KB regarding the problem that we have all known about for
ummm… how long is it already :) ?

Cannot launch vSphere Client after installing in Windows 7 or Windows 2008 R2


I guess that all the traffic that Joep Piscaer was getting on his blog post on the subject will start to slow down now.

VMware – we are still waiting on a permanent fix for this!

Patching your ESXi Host – Without vCenter

VMware Update Manager is the Enterprise tool for Patching your ESX Hosts and for some also the tool used to patch your Windows / Linux Guests as well.

This is all fine and dandy, but what is you do not have all of your ESXi hosts connected to your vCenter?

Why would you so that – you may ask? Well in my environment, we have several labs that are running their Environment on a ESXi Whitebox,with the free ESXi License. No vCenter, and central management is not that easy.

Now you could update all of these hosts with the VMware infrastructure Update Client and I showed in a previous post – how you can upgrade from ESX3i to ESX4i with this tool. To update the patches is more or less the same.

Today I was asked to automate this process for a number of ESX3i hosts that were all at
build 123629 (ESX3i U2) and update them to build 199239 (ESX3i U4).

I did this with the GUI client, but it was too cumbersome, so I looked for another method.

I downloaded the package from VMware. This contents of the zip file are below.

image

As you can see the contents of this bundle contain 3 separate Updates.

(The three ESXi patches for Firmware "I", VMware Tools "T," and the VI Client "C" are contained in a single offline "O" download file.)

After Unpacking the 3 patches I was left with this:

image 

In each folder there is an exe file:

image

The Command Syntax:

C:\ESXe350-200910402-T-BG>remoteInstall.exe

Allowed options:
  -? [ --help ]         Display program usage

  -h [ --host ] arg     ESX host name
  -u [ --userName ] arg User name
  -p [ --password ] arg Password

So all you need to do is:

remoteInstall.exe –h <HOSTNAME> –u <USERNAME> –p <PASSWORD>

While the patch is installing the output on the screen is is the progress:

progress: 0
progress: 5
progress: 7
.
.
.
progress: 100

Run this for both 3 patches, reboot your ESXi Host, and you are done

To automate the process I created  a small batch file update.bat (I did not use Powershell – because not all the end-users have Powershell installed – YET!! )

echo off

set host=%1
set user=%2
set password=%3

if "%1" == "" goto error
if "%2" == "" goto error
if "%3" == "" goto error

for /f "tokens=*" %%i in ('dir *. /b') do (
cd %%i
echo Installing patch %%i .....
remoteinstall.exe -h %host% -u %user% -p %password%
cd ..
)
echo Patching Complete !!

:error
echo ==========================================================
echo = Missing command line parameter!
echo ==========================================================
echo =
echo = Command Line Parameters:
echo =         update.bat hostname/ip username password
echo =
echo = Example:
echo =         update.bat 1.1.1.1 root 123456
echo ==========================================================
goto end

:end

 

The script accepts 3 parameters: user host password, for example:

update.bat myesxhost root 123456

Reboot your host – and yep you are upgraded.

And thanks to Dave Mishchenko for the information.

2009-11-09

A useful statistics tool

>

This is not a ESX or a virtualization tool but for those who are interested in tracking their blogging traffic.

First thing is to install Google Analytics on your blog. I will not go into the details of how that should be done, Wordpress and other blogging platforms have plugins to do this for you.

Once you have it defined, you can either:

  1. Go into the site every day and check you statistics.
  2. Set up a report that will land in your inbox at your specified interval
  3. Use Polaris.

So what is Polaris?

Polaris is a cross-platform desktop widget for Google Analytics. With 8 standard reports it's the easiest way keep your data always instantly available. The rich interface and swift navigation make it a pleasure to use.

image

Polaris is a free download for everyone who manages one website profile. For those who are managing multiple profiles, the upgrade to the full version is $15 / year. Both the free and the full version will get regular free updates with new functionality and features.

A great app, free and is of course cross-platform. Highly recommended!