Category Archives: ESXi

Using the Free vSphere Hypervisor? VMware Wants to Hear from You.

Interesting that I stumbled on this right after I posted a How-To for the free vSphere Hypervisor.

Mike Adams over at VMware wants to know if you are using the free vSphere Hypervisor. If so, he would like you to complete a very short survey so he can understand how you are using it. Check out his post on the VMware blog.

ESX is Going Away – How to Migrate to ESXi

If you didn’t know it yet, VMware announced a while back that future releases of VMware will not include the “traditional” ESX Server. From their site:“VMware vSphere 4.1 and its subsequent update and patch releases are the last releases to include both ESX and ESXi hypervisor architectures. Future major releases of VMware vSphere will include only the ESXi architecture.”

If you are in a “24/7/365” shop then the applications running in your private cloud should currently be in virtual data centers (vDC) that are contained in DRS/HA clusters and the migration can be completed with no downtime to the applications. However, there are still other systems, such as development and test systems or possibly some minor infrastructure services applications that may not benefit from vSphere’s availability features. I know many people have scheduled outages, shutdowns, etc. during the upcoming holidays. It may the best time to migrate to ESXi…

A Little Bit About ESXi 4.1

I have actually been pushing ESXi since version 3.5. In every plan and design engagement where I have been involved, I have always started with ESXi as the version to be used unless there was a compelling reason NOT to use ESXi. I think the only thing right now that requires ESX is HP’s Matrix. I have yet to find anyone that uses that behemoth. There have been many improvements to the features of ESXi since version 4.0. VMware has a nice “Why ESXi” web page to explain these new features. The biggest thing is that ESXi has a smaller footprint and has fewer security updates needed than the “traditional” ESX Server. For instance, the latest round of patches has 11 for ESX and only two for ESXi. The small footprint allows ESXi to be installed on a USB stick, an SDHC or just on less disk space. There is also a nice VMware KB article explaining all of the differences between the current version of ESX and ESXi.

Here are some other things that I like about ESXi:

Treat the ESXi Server like it is an appliance

The ESXi installation should be treated as firmware. It is even called firmware in Update Manager. This means that there is no actual upgrade. The firmware is installed fresh every time. This also brings the idea of having stateless servers at some point in time. More on that later.

Kickstart Scripting

Really, there are three methods for a mass deployment. One is to use Host Profiles, but this requires manual steps and the extra costs associated with Enterprise Plus licenses. A second method is to install a base install of ESXi using the method I outlined in a previous post and then use PowerCLI or the vSphereCLI to customize the server. The new preferred method is good ‘ol Kickstart.

Several Boot Options

Now, you can boot from local disk, SAN Disk, a USB Key or an SDHC Card. One of the arguments some have against booting from USB or SDHC is the dreaded single point of failure (SPOF). My answer has always been that HA will cover this. And, if you think about it, an internal array controller is a SPOF as well. But, now you can boot from SAN if you wish. Just remember to create a space for logs and core dumps if you go the route of USB or SDHC.

Active Directory Integration

ESXi servers can now become members of a Micro$oft Active Directory Domain so that administrators can authenticate to the directory. Setting this up is done through the vSphere Client under the configuration tab. It appears that you can call the administrators group anything you want in AD as long as it is “ESX Admins”. Maybe that will change in future versions.

Hardware Monitoring

You can just do away with SNMP monitoring and use the Common Information Model (CIM) providers. The stock version comes with some basic CIM monitoring, but the major hardware companies provide custom baked versions of ESXi with OEM-specific CIM providers for more granular monitoring. Rather than setting up your monitoring software for SNMP, you just point it to the ESXi server and set it up for CIM or WBEM. If you are not using a monitoring server, you can use the vCenter server to handle alerts and alarms.

Enhanced Tech Support Mode and vCLI Operations

Using Tech Support Mode, formerly known as “Unsupported Mode,”  is now supported. You can log in as an administrative user and all commands run in TSM are logged. The biggest reason many people went into TSM was to kill a stuck VM. There is now a vCLI operation that will allow this.

The Migration Path to ESXi

Here is the “Super Dave Method” for migrating to ESXi. Like I mentioned before, if you have at least ESX Clusters with vMotion enabled, this will be a relatively painless process, with no downtime. Obviously, if you are also upgrading from a previous version, there will be reboots required for VMware Tools and possibly to upgrade the VM Hardware to version 7.


Yeah, you heard me. Take some time to familiarize yourself with the differences between ESX and ESXI by setting up at least one test system so you can hammer it before changing your production systems. The nice thing is that you don’t need hardware to do this. You can set up a VM and run ESXi as a VM.

vMA, vCLI and PowerCLI – Oh My

If you are already familiar with doing things from the ESX console, the vCLI will be very familiar to you. If you are a Winders guy, you should already know PowersHell, so the PowerCLI will be familiar. Also, get to know the vSphere Management Appliance (vMA). The vMA is a RHEL5 based VM Appliance that is pre-configured with the Perl SDK and vCLI. It also includes vi-fastpass command, which allow you to pass authentication through to the hosts and to vCenter. It also includes vilogger, which allows you to create a (very) poor man’s syslog server. I think you are better served using Splunk! or phplogcon. Either will allow you to parse the syslog entries eisier if you need to do any troubleshooting or forensics. If you want a nice guide to the vMA, head over to the vGhetto for some nice tips and tricks. If you are already a Linux shop and RHEL is not your cup of tea, then you can bake your own vMA with the Perl SDK and vCLI, but you will lose the vi-fastpass and vilogger capabilites. vi-fastpass is not a huge loss and you can set up your home baked vMA to include syslogd and phplogcon.

Create a Kickstart Script

The most efficient way of performing an installation on more than one server is to script it. VMware now supports using Kickstart. Test your KickStart script on the test ESXi server, weather it is physical or virtual, you should be able to test most of the functionality. Check out the nifty “DryRun” setting too. I am not going to rehash what has already been done or said regarding Kickstart, so here are some decent links:

Take a look at this VMware Labs Fling if you want to create a nice deployment server that automates the ENTIRE process. This brings up the idea of having stateless ESXi servers. As each one is inserted into the vDC, ESXi is automatically set up for you. All the “important stuff” is stored on shared disk.

If you decide to go the route of using PowerCLI, take a look at this nice post.


Back up EVERYTHING! ‘Nuff said? There is no better feeling when the shit hits the fan than to have a good backup. Make sure you back up the vCenter database and capture the configuration settings of all of the ESX servers. If you have Enterprise Plus, capture a host profile for each server. Take screen shots of the configurations settings in the vSphere Client. Include networking and all of the tabs in the vSwitches. Include storage and all of the tabs of any iSCSI initiators. Don’t forget the IQNs! Check the advanced settings for any tweaks. Document the whole thing.

Pick the First Victim and Move Your VMs

If you are using DRS and HA, disable HA. Then place the first victim host in Maintenance Mode. This should automagically move the VMs to other hosts in the cluster if DRS is working. Or you can manually (*GASP*) vMotion your VMs. If they are on local storage, set up an iSCSI server – OpenFiler is free. You can then use Storage vMotion to move the VMs to the iSCSI storage.

Remove from inventory

Yep! If the server is a part of a cluster, remove it from the cluster. Before removing it from inventory, unassign the license. Then, remove it from inventory. Once it has been disassociated from the vCenter Server, shut it down.

Now is a good time to update the BIOS, device firmware, etc. Blow out the dust. Perform some hardware TLC. If you do not trust yourself, disconnect it from the SAN. If you are using software iSCSI initiators or NFS, no worries because you will need to reconfigure this stuff after the install.

Install ESXi

Now is the time to actually perform your first ESXi installation. Go ahead, we’ll wait.

Post Configuration

Once ESXi is installed, you can add it back to the vCenter Server, add the license and perform any post configuration steps like apply the host profile or run your PowersHell script for post configuration. Once the post configuration is completed, confirm that all of the settings are correct and match what you documented previously. If all is good, add it back to the original cluster. If you want to set up Enhanced vMotion Compatability (EVC) and it’s not currently set up in the original cluster, you can create a new cluster for the ESXi Servers with EVC enabled.


Repeat the process for each server. If you decided to create a new cluster, start moving VMs to the new cluster.

More Tips

If you are installing ESXi to a USB Stick or an SDHC card:

  • Make sure the stick or card is supported by the hardware manufacturer.
  • Set up a syslog server. This is very important.
  • If you don’t set up a syslog server, make sure you have a swap partition on the centralized storage. If you do not do this, the logs are deleted on reboot. Then set up a syslog server.

Make sure that during the test phase you understand how to set up authentication, syslog settings and nay required custom settings. Also make sure you know your way around the DCUI and the Tech Support Mode console.

Creating an Automated ESXi Installer

Back in the summer, I saw Stu’s Post about automating the installation of ESXi. I was reminded again by Duncan’s Post. Then, I found myself in a situation when a customer bought 160 blades for VMware ESXi. With this many systems, it would be almost impossible to do this without mistakes. I took the ideas from Stu and Duncan and created an ESXi automated installer that works from a PXE deployment server, like the Ultimate Deployment Appliance. I took it a step further and added the ability to use a USB stick or a CD for those times when PXE is not allowed. The document below is a result of it.

This is a little different than the idea of a stateless ESXi server, where the hypervisor actually boots from PXE. This is the installer booting from PXE so that the hypervisor can be installed on local disk, an internal USB stick or SD card. You could also use it for a “boot from SAN” situation, but extreme care should be taken so you don’t accidentally format a VMFS disk.

As always, if anyone has comments, corrections, etc., please feel free to post a comment below.

The document can be found here -> Creating an Automated ESXi Installer


The ability to use an automated, unattended installation routine for a hypervisor is necessary whenever it is deployed to multiple systems or is done frequently. Automated installations help avoid a misconfiguration caused by human error, which become common when repetitive tasks are performed.  Because the “traditional” version of VMware ESX Server contains a Red Hat Linux based console operating system, we have been able to leverage kickstart scripts for automated installation. With the ESXi hypervisor, much of this functionality is not available because of the smaller footprint.

This document explains how to set up ESXi with little intervention. The modifications explained here can be used to deploy ESXi using a PXE server. In our examples we will use the Ultimate Deployment Appliance, but these methods will also transfer to such commercial packages as HP Rapid Deployment Pack, Altiris, or even a home grown PXE server. The modifications can also be used for deploying ESXi using a USB stick or a customized CD.


  • ESXi Server Installable The ESXi CD image can be downloaded from the VMware site, however using a systems management and monitoring server, such as HP SIM or Dell OpenManage is highly recommended. Since there are usually vendor specific CIM providers to enhance the monitoring capabilities, some vendors will provide a customized CD image with the CIM providers. These additional CIM providers will also allow for more information to be displayed in the hardware sections of the vSphere Client. A search for “ESXi” on the HP and Dell sites produced links to the latest customized images.
  • Deployment Server A deployment server will allow for a controlled, automated installation of the ESXi Server software. The ability to handle multiple operating system installations is also desired. The ability to provide PXE and DHCP services is required as well. Most times, the deployment server will be running PXE services and TFTP. The DHCP services may be running on a different server in an enterprise. This document does not explain how to set up a separate DHCP server. For this document, we will be using the Ultimate Deployment Appliance (UDA) version 2.0 (beta).
  • Virtualization Software The UDA runs as a “Virtual Appliance,” which is a pre-configured virtual machine. It will run under VMware ESXi (available as a free or licensed instance), VMware Workstation (available for purchase), VMware Player (free) or VMware Server (free). In this document, VMware Workstation is used.
  • Optional software Although no additional software is required when using the UDA, you will need additional software if you plan on using a USB stick or if you plan on creating a customized CD image:
    • VMware Converter If you plan on using ESXi or Server to host the UDA, VMware Converter can be used to import the virtual appliance.
    • Syslinux In order to make a bootable USB stick, you will need the syslinux utility. This utility is available for Linux and Windows. The UDA does not include it. As an alternative, you can use the unetbootin utility.
    • CD Imaging and Burning In order to create a bootable CD image, you will need software to create the CD image (mkisofs) and then software to burn the image to the CD media (cdrecord). The cdrtools project includes versions for Linux and Windows. Most Debian versions of Linux, such as Ubuntu, come with the cdrkit, which uses genisoimage for imaging and wodim for burning.
    • Linux Desktop If you look at the contents of the ESXi CDs using Windows (Windows 7 was used), you may see all of the files listed using all capital letters. Since the ESXi software is based on Linux, all file operations are case sensitive and expect the files to be all lower case. This may cause errors when attempting to create the automated installer. For this reason, a Linux desktop is recommended. For most of the operations, UDS may be used. The only missing software on the UDA is syslinux. For a feature rich Linux desktop, Ubuntu is recommended. A few pre-configured Ubuntu Desktop virtual appliances are also available.


Once you have a hypervisor installed you will need to configure the server and add it to vCenter in an automated fashion. Look for a future doc covering this. For now, check out these resources for post install configurations: