Monthly Archives: April 2009

Get Hal's PowersHell Book Here!

This is just a shameless plug to try selling books on DailyHypervisor.com…As you already know, Hal Rottenberg has written a book called “Managing VMware Infrastructure with Windows PowerShell TFM

Get it here:

Managing VMware Infrastructure with Windows PowerShell TFM

Installing VMware ESX 4 in Text Mode

There are many reasons to install VMware ESX in text mode. The main reasons I use text mode are that it seems quicker for me and text mode responds better when using remote console connections, such as iLo, DRAC or console over IP. Previous versions of VMware used a text mode that incorporated Anaconda and was very similar to the text mode for RPM based Linux distributions. The new text mode in ESX 4 is VERY rudimentary when compared to the earlier versions. Hoever, it performs very well and is fairly straight forward to use.

The text mode installer uses simple lists of choices. Usually, 1 is for continue or to answer yes. Some items will have more than one choice. Here is a screenshot:

esx4-001-2009-04-27-173430

The console OS truly appears as a VM in this version. You must create a datastore and then a VMDK that represents the COS. A disk of sufficient size will be required for this. My first attempt, using an 8GB disk failed. My second attempt, using a 10GB disk was successful.

You can download a doc outlining text mode installation HERE.

VI4 Certification News

From the VMware Certification Site:

Certification News:

  • VMware Certified Professional on vSphere™ 4
    With the launch of vSphere™ 4, a new certification will be available. The VMware Certified Professional (VCP) on vSphere™4 beta exam will be available 30 days post GA. Candidates eligible for the beta exam will be contacted directly by VMware.

    There are four possible paths to acheive VCP on vSphere™ 4

    1. If you are NEW to VMware
      • Attend the VMware vSphere™ 4: Install, Configure, Manage course (first courses available in late June 2009) OR attend the VMware vSphere 4: Fast Track (available in Q3)
      • Take and pass the VCP on vSphere™ 4 exam
    2. If you are currently a VCP on VMware Infrastructure 3
      • Take and pass the VCP on vSphere™ 4 exam. This option will only be available until December 31, 2009. Beginning in 2010, VCPs on VI3 must attend the VMware vSphere 4: What’s New class in order to upgrade.
    3. If you are currently a VCP on ESX 2.x
      • Take and pass the VCP on VMware Infrastructure 3 exam
      • Take and pass the VCP on vSphere™ 4 Exam. This option will only be available until December 31, 2009. Beginning in 2010, VCPs on VI3 must attend the VMware vSphere™ 4: What’s New class in order to upgrade.
    4. If you are not a VCP on VI3, but have attended one of the prerequisite classes (Install & Configure; Deploy Secure & Analyze; or Fast Track).
      • Take and pass the VCP on VMware Infrastructure 3 exam OR attend the VMware vSphere™ 4: What’s New course.
      • Take and pass the VCP on vSphere™ 4 Exam.

    …more info

FLASH: ESX 4 Console OS is REALLY a VM this time!

While I was setting up ESX in text mode for my next blog post, I discovered that the installation sequence first creates a VMFS file system and then creates a VMDK file for the console OS. I confirmed it in the VIC. Here is a screen shot:

2009-04-29_174111

Click to enlarge image

I also noticed that the logs are now in a separate directory:

2009-04-29_174129Click to enlarge image.

Running VMware ESX 4 RC in a VMware 6.5.2 VM.

I just set up another quick VI4 lab on my laptop for the purposes of capturing screen shots and testing some things out. I was worried because I was not able to start VMs in this lab using ESX 4 Beta 2, but everything is fine again! Here is a screen shot of a Winders 2003 VM running inside an ESX 4 RC VM which is running inside of Workstation 6.5.2 on an Ubuntu machine.

vm-in-vm

Click on the image for a full-size view.

My VMX settings were from a post on VMTN when I was trying to get ESX 3.0.x to run on a WS 6.0.  Actually, XTraVirt came up with the solution originally.

Well, my VMX has not changed MUCH since then. I only added some parameters for sharing SCSI disks so I don’t need an iSCSI server. I found THAT information on Duncan’s Blog.

##################################################
# Start DAC Customization

guestOS = “other-64”

monitor_control.restrict_backdoor = “true”
# monitor_control.virtual_exec = “hardware”
monitor.virtual_exec = “hardware”
monitor_control.vt32 = “true”

# REQUIRD FOR USING NTFS DRIVES WITH LINUX HOSTS
mainMem.useNamedFile=FALSE

# For SCSI disk sharing
disk.locking = “FALSE”
diskLib.dataCacheMaxSize = “0”
diskLib.dataCacheMaxReadAheadSize = “0”
diskLib.dataCacheMinReadAheadSize = “0”
diskLib.dataCachePageSize = “4096”
diskLib.maxUnsyncedWrites = “0”

bios.bootDelay = “5000”

ethernet0.present = “TRUE”
ethernet0.connectionType = “custom”
ethernet0.wakeOnPcktRcv = “FALSE”
ethernet0.vnet = “/dev/vmnet3”
ethernet0.virtualdev = “e1000”
ethernet1.present = “TRUE”
ethernet1.connectionType = “custom”
ethernet1.vnet = “/dev/vmnet3”
ethernet1.virtualDev = “e1000”
ethernet1.wakeOnPcktRcv = “FALSE”
ethernet2.present = “TRUE”
ethernet2.connectionType = “custom”
ethernet2.vnet = “/dev/vmnet3”
ethernet2.virtualDev = “e1000”
ethernet2.wakeOnPcktRcv = “FALSE”
ethernet3.present = “TRUE”
ethernet3.connectionType = “custom”
ethernet3.vnet = “/dev/vmnet3”
ethernet3.virtualDev = “e1000”
ethernet3.wakeOnPcktRcv = “FALSE”
ethernet4.present = “TRUE”
ethernet4.connectionType = “custom”
ethernet4.vnet = “/dev/vmnet3”
ethernet4.virtualDev = “e1000”
ethernet4.wakeOnPcktRcv = “FALSE”
ethernet5.present = “TRUE”
ethernet5.connectionType = “custom”
ethernet5.vnet = “/dev/vmnet3”
ethernet5.virtualDev = “e1000”
ethernet5.wakeOnPcktRcv = “FALSE”

ethernet0.addressType = “generated”
ethernet1.addressType = “generated”
ethernet2.addressType = “generated”
ethernet3.addressType = “generated”
ethernet4.addressType = “generated”
ethernet5.addressType = “generated”
# End DAC Customization
#####################################################

My next posts will involve installing ESX 4 in text mode and some very interesting findings during that install….

VMware Partner Exchange Technical Notes

Below are some notes that I took from the VMware Partner Exchange technical sessions that I attended.  I left off the BCDR Workshop because it will become a separate post.

Upgrade and Migration Tips:
This session was conducted by Mustafa Kahlil, who is probably THE senior SE at VMware. He has over 10 years with the company and started around ESX 0.9. The session centered around a group of flow charts that followed a nice decision tree for upgrade or migration to VI4. The flowchart will provide everything you need for a upgrade / migration engagement. Some highlights: “Upgrade VMotion” will perform a combination VMotion from ESX 2.5 to ESX/ESXi 4 and a Storage VMotion from VMFS 2.x to VMFS 3.x. A VMotion licence will be required. This will allow direct MIGRATION from ESX 2.5 to ESX/ESXi 4. In order to UPGRADE, the latest update build of ESX/ESXi 3.5 will be required. A “vSphere Update Utility” (AKA VMware Infrastructure Update Utility) will be used to update ESXi if VUM is not used. VUM will be the easiest path. The utility will only be able to update one host at a time, but could be scripted to perform a chain of updates. THERE WILL BE NO UPGRADE PATH FOR VMs ON NFS! Only migration will be used for NFS VMs. The most interesting thing Mustafa said was

“Eventually, no service console”

Remember this in your Plan and Design engagements for VI3. I have always been a supporter of ESXi over ESX.

Performance Best Practices for ESX:
This was, by far, THE BEST session. It was two hours of drinking from a fire hose. I took three pages of notes and they didn’t get through all of the slides. Next week, they are expecting to release an updated performance whitepaper on the VROOM! site. Here are the main things that should be considered for performance:

First and foremost, VM performance has little to do with the ESX configuration tweaks, but has a LOT to do with VM configuration tweaks. The main points were this – Understand the application profile, choose the platform wisely and tune the VM settings. All of the performance information below assumes enough CPU and RAM are provided to an app.

Most CPU intensive apps virtualize very well. Most memory intensive apps virtualize very well. As stated before VMware demonstrated a RHEL/ORA VM that could handle 8x of Visa’s online transactions.

Network usage – between 1 and 16 Gbps will virtualize with proper VM tuning, Over 16Gbps will not. (40Gbps in VI4)
Storage I/O – between 10-100k will virtualize with proper VM tuning. Over 100k will not (200k in VI4)
Anything with more than 8CPUs or 255GB RAM will also not virtualize.

See the doc -> http://www.vmware.com/pdf/asplos235_adams.pdf

Use Intel VT / AMD RVI. The newer chips also include hardware based paging for memory, which takes a load off of the hypervisor.

Use TSO and Jumbo Frames for networks. JF is disabled by default and must be enabled on all devices involved – NIC BIOS, vSwitch, VM OS, pSwitch. VI4 will support JF in iSCSI as well. pNICs should also handle 64bit DMA and multiple scatter/gather elements for the frame. Also, force speed and duplex, separate traffic, use teamin, etc.

Configure storage properly – spindle count, hot spots, LUN layout, etc. Use VMFS instead of RDMs. Use 4k I/O for best performance over 16k and 64k.

VM setting:
Choose the proper OS. This determines the proper monitor type, optimal devices, etc.
Use 64bit OSes for high memory usage
Enable large pages in OS and app (default is disabled)
most apps do not scale well beyond 4/8 CPUs with the exception of RHEL/ORA

See the doc http://www.vmware.com/files/pdf/consolidating_webapps_vi3_wp.pdf

“STORAGE QUEUE SETTINGS ARE ALREADY OPTIMIZED”

They should not need to be tweaked! Kernel latency MAY indicate it is necessary to tweak queues.

Perform adminitsrative tasks (new VMs, clones, etc.) during off hours. These produce storage performance hits.

VMware Partner Exchange Notes – Keynotes

Here are some breif notes from VMware Partner Exchange. I will also post about some of the technical sessions. I am not going to regurgitate the keynotes. The content will be available soon on Partner Central and there are several blogs that have plenty of information from the keynotes. I will however provide some highlights:

Partner Central and Partner University will soon be revamped. The accreditations will be changing and will require a certain number of accredited VCPs before the company can get an accreditation. The categories will be similar to our practices, such as infrastructure virtualization, desktop virtualization and BCDR. If you go to partner central now and click on the partner university link, you will see a little bit of what the changes will be. There is also plenty of web-based, self-paced training. On line tests are available so you can receive accreditations for many different products, most are jumpstarts and plan and design related.

VMware’s obvious desire is “100% virtualized.” Their primary focus will center around cloud computing with an initial push for the internal cloud as many see challenges with getting acceptance for the external cloud. Private clouds will eventually bridge the gaps between the internal and external clouds. Much of this information is already available on VMware’s main site.

The software surrounding VI4 took around 3 million engineering hours to develop. It includes great improvements on resources that will be available to the VMs. The resources will be increased to 8 vCPUs per VM, 256GB RAM per VM, 40 Gbps network throughput per VM, and 200,000 storage IOPS per VM. vCenter maximums will increase to 3000 VMs / 300 Hosts. There will also be a capability for linking up to 10 vCenter servers with a centralized search function.

A new function centers around host profiles, which works similarly to VUM. It establishes configuration baselines for an ESX / ESXi host that includes such things as network, security, storage and NTP settings. A host can be scanned for compliance and remediated with the baseline. The BIG “however” is that it will require “Enterprise Plus” to enable host configuration controls and distributed switches. This will carry a $600 price tag and is not ala carte.

Using ESX4 allowed for 85% native performance on 8-way RHEL/Oracle servers in spec performance tests. The amount of transactions (I forget how many) was 8x the number of Visa’s current transactions.

Out of the gate, vSphere will offer optional components surrounding security, BCDR and networking. Additional vSpere components will become available “over the summer.”

Questioning SaaS

I was torn on whether or not to post this rant, but then I read a post that made my head spin….

First, there was the “Great Gmail Outage of February 2009“. There are constant Twitter outages as it grows in popularity and the servers struggle to keep up. Just last week, Yahoo Mail and Hotmail users were suffering through outages. I read on one site “Although the timing of the incident means that UK customers are unlikely to have been affected, the news will add to those doubts some users have over the software-as-a-service model.” This is the post that nudged me into posting this rant. I have had a few Hotmail accounts since 1998 and have had occasional access issues through the years, before I even knew what SaaS meant. My question is this: So What?!?

How can you doubt Saas because your free email is down? Free is free. You get what you pay for. I read that Google has offered credits to the paying GMail customers, and that is the proper thing to do. But how can executives whine because their GMail/Hotmail/Yahoo is off line when they don’t pay for it? Why are they not paying for a business email service?  I have worked for a few companies that have used “ousourced” paid email services – the REAL model for SaaS. I have had scheduled outages during hours when I am sleeping.

The fact is that Saas is here to stay and it is increasing in value and popularity. Yes, Google is leading the way with their free apps.  Saas is a piece of Cloud Computing. Check out this video explaining Cloud Computing in Plain English:

Its VMware Patch Friday…

VMware just released a round of patches. Go get some -> http://support.vmware.com/selfsupport/download/

Obviously, if you have VMware Update Manager set up, he’ll do all of the work for you. :o)

Ken's Networking Tips…and Cracking the ESX Root Password

First…Ken Cline started a blog about a month ago. It has some nice tips for networking, so check it out. My eyes were opened after reading his post about accepting default settings. I know the post is almost a month old, but I have that reading narclepsy thing. It is still a very important thing to read. My philosophy is similar to Ken’s:

Just because you CAN do something does not mean you SHOULD do it.

I guess I picked up the habit of creating a slightly complicated way of connection two pNICs to a vSwitch for redundant connections for the Management Network and VMotion Network. I think this came from the old school (ESX 2.x) way of doing things. Ken’s methods are far easier to set up and manage.

Now…on to cracking the root password on an ESX server….

VMware just released a KB article about how to change a “forgotten” root password. WOW. Pretty simple.. Anyone who knows a smidgen about Linux could have helped here. It was even a part of the RedHat test back in the day when I took it.

First off, there should be no reason to need this if proper security and change control practices are followed. Passwords should be changed regularly and they should be kept somewhere. That somewhere should be secure. ‘Nuff said.

Second…And this is the biggie…. If you follow security best practices, this method will be rendered USELESS. The boot loader (Grub in this case) should be secured properly to protect from this sort of attack. Yes, I used the “A” word here. Unauthorized entry into single user mode is an attack. If Grub is secured properly, you will need to know a password to enter the append or edit modes. This password is just as important as the root password and proper security and change control practices should also be followed here.

OK…Now your server boot loader is “secure”.  The root password is fracked and you don’t know the grub password… Now what? Enter the rescue CD. A live Linux CD, like DSL or Knoppix will allow you to boot into a Linux session and mount your boot partition and edit the grub.conf file. Now, you can boot the server and append the boot loader line to exit single user mode. Or, you can mount the root partition, chroot and change the root password that way. It is less than simple, but we are talking about “attacking” a server and changing the root password. How do I prevent the “Live CD Attack”? Use a BIOS password and set the server so it does not boot from external media. This password should also follow the security and change control practices.

Next..”cracking” the BIOS password. In a few words: DIP switch 5. Most servers have a bank of DIP switches. Flip one of them and the BIOS password is disabled. How do you avoid this? Lock the server. I have and HP key that opens any HP cabinet and a few Dell keys that do the same thing….

I am going to stop now. I am not a computer security expert by any means. I just use common sense. Three thousand years ago, I worked for an alarm company. I could get into any “secured’ alarm cabinet and shut them down. My philosophy back then was:

Locks are for honest people…

Back to the original intent of the password thing. Use common sense. The method VMware posts for resetting a lost root password should not be possible. It all falls down to common sense and the use of good security and change control best practices. Many attacks come from within. Locks and security measures will only slow someone down and hopefully trigger that alarm system to notify someone of the attack.

Hal's New PowersHell Book!

If you want to learn something about PowersHell and the VI Toolkit for Winders, the community forum is the best spot right now. Hal Rottenberg is one of the pillars of that section of the forums. Always glad to help figure out your code when things are not working and always ready to explain what the heck is going on with it. I think he taught me most of the things that I know about the VI Toolkit.

Well he has written a book called “Managing VMware Infrastructure with Windows PowerShell” Stop over to his blog and pre order the book. I am sure it will become a valuable asset in your tech library.

What are you waiting for? Get out your credit card!

The Open Cloud Manifesto

The Open Cloud Manifesto was released the other day. The list of supporters is pretty impressive and the non-present are typical. I actually read the manifesto last night on my Blackberry during my daughter’s piano lesson. It was a nice read, even though the site does not have a mobile format.

The idea of cloud computing is apon us. Are you ready for it?

What Would You Like to See Changed in the VMware VM Backup Guide?

As you already know, I posted a generic VCB Proven Practice Guide on VI:OPS. I refer people to this doc frequently.

A recent community discussion regarding the VCB Documentation was visited by a VMware employee or two and the question was posed: “Could you suggest areas of improvement for this guide.”I posted a lengthy response this morning. This might be your chance to comment on the VCB documentation. I am posting the link in hopes of you responding to the question. Hopefully, the comments are considered.

Go over to the VI:OPS site and suggest changes to my doc as well!

Fixed: VMware Tools status shows as not running after running VMware Consolidated Backup

A while back I mentioned that VMware Tools would appear to change to a not running status after a VCB Snapshot was taken. Vmware said a fix would be forthcoming in ESX U4. VMegalodon posted on the communities this morning that he is running VC 2.5U3 and ESX 3.5 U4 (Which is probably a bad combination…) and the VMware Tools issue appears to be corrected.

So, what are you waiting for?? Get to upgrading!

Thanks VMegalodon!