Let me give you a real case scenario. (I am not saying that any of the implementations in this example are best practice)
Default rack that was installed was a 42U rack.
Each rack was populated with no more than 16 Servers. (This was the most that they were willing to lose in the case of losing a full rack and also due to cooling issues per rack they were leaving 1U empty between each server).
Each rack has a 16 port LCD KVM keyboard tray (1 for each server).
Each rack has 24 1Gb Ports and also a 48port 10/100mb for backup network and management ports.
Up until now each Server by default had 1 NIC connected to a 1Gb port and the backup to the 100mb port - which left them with 16/24 1Gb ports populated.
Enter ESX.
Each ESX server has by default 6 NICS
- vmnic0 + vmnic5 - SC + Vmotion
- vmnic1 + vmnic3 - Iscsi/NFS storage
- vmnc2 + vmnic4 - VM network traffic
Out of the 16 servers in each rack 2 ESX servers in a rack (the most they would risk is two hosts per rack)
Quick math again.
2 ESX hosts x 6 ports = 12 (All port have to be 1Gb)
OK 14 more servers in the rack - but one sec, there are only 12 ports free - which means that you can only put in 12 more servers.
The options were:
- Run more 1GB cables to the Network Infrastructure - more switch ports which meant additional cost
- Reduce the number of servers per rack to 14 servers per rack (not 16) - which meant 2 KVM ports were wasted and more empty U's in each rack.
- Run some production servers not on 1Gb infrastructure
- Deploy 10Gbe - cost for 10Gbe NICS and modules for the Backbone infrastructure
What happened here? The racks were cabled several years ago - before Virtualization was something that they had in their datacenter - and they had what they thought more than enough place to grow in their network ports, and they did until they started to put in ESX servers with 6 NIC's each.
Of course all of the options above have a cost with them - be it a cost that you have to lay out immediately or cost of not utilizing the resources they had to the full extent (rack space + KVM).
What was decided in the end, you may ask - well I do not have a definitive answer to that yet - because it is still under deliberation - as to which option they will choose.
What can you learn from this experience?
- You should review you datacenter infrastructure at regular intervals. What was suitable for you 2 years ago is not necessarily the same today.
- Bringing in a new technology will definitely have some benefits - but it will also have its caveats as well.
- Virtualization is not only something that touches one aspect of your infrastructure (be it your initial savings on costs of Hardware / Operating System Licensing) it affects everything:
- Network - what ports and how many
- Storage - is your storage fast / big enough?
- Hardware - do you scale up / scale out
- Operating Systems - Licensing models will change
- Backup / Restore - will your system suffice?
- BCP / DR - should you use new technologies (SRM)?
- IT Procedures - Server provisioning - and lifecycle
- Manpower - Maybe your headcount will shift according to your changing needs
- Cooling - depending on your hardware your cooling needs will change
- Electricity - ditto
- Maintenance fees - for hardware will change - and software as well
- OPEX / CAPEX - VMware Licenses - MS licenses - Hardware contracts
- Rack Space - how much will you need in a rack?
- User Education - they will get exactly what they pay for and do not need a gazillion cores :)
As you can see the list is long - and I am actually not sure that I have all the elements listed here.
I would be interested in hearing from you all if you have any additional elements that this will affect and your thoughts in this article.