Friday, April 17, 2009

VLAN / Portgroup replicator tool

This is not a new tool (ITQ Infrastructure Client), but it is still very useful. Let's say you have to setup six hosts in a cluster and you have to configure two vSwitches on each host with several portgroups or VLANs on each switch. This can take quite some time if you're scripting the post confirguration. And there are plenty of room for error as each host is configured manually.

With this tool, you only need to configure the vSwitches and VLANs on one host manually. Then you can export the network configuration and apply it to the remaining hosts all at once.

With VI4 this feature will be built in with "Host Profiles" but until then, this tool can be used.

Wednesday, April 15, 2009

Master VCP certification is on the way

The VCDX certification is rather complicated to acquire. You have to travel to take the exams (enterprise administration and design exam) and you have to submit a written report. This is all a bit too much of a hassle for most people. But now apparently, according to a post on Eric Sloof's site, there's a new VCP Master on the way which is obtained with a VCP and then only the Enterprise Administration exam.

NLB in VMware

There's a lot of posts in the forums about Microsoft NLB (Network Load Balancing), and it can be quite confusing to figure out what's what. We had problems with it for months before we got it fixed. One of the reasons that it took so long is that there are two ways to go about it and one way is to tweak ESX vSwitch in an ugly way an the other is nice and neat but involves the involvment of the network department - I'll describe the latter solution.

First of let me state: NLB works just fine in VMware. It is not necessary to make any changes on the ESX server or in the vSwitch.

The following steps should be in place:

1. Install and configure NLB on your Windows server like you would in a physical setup.
2. NLB should be configured for multicast
3. A static arp entry needs to be added to the physical layer 3 switch or router that acts as default gateway (note the default gateway from within Windows) for the NLB nodes. If it's a Cisco switch, then the command will be like this (if it's not Cisco, then it may work without adding the arp entry...):

arp <ip of nlb cluster> <mac of nlb cluster> arpa

Adding a static ARP entry to the ARP table means that you're associating a specific IP address with a given multicast mac address. This is not automatically allowed in the Cisco switch/router and therefore such requests are dropped unless manually added.

If the default gateway happen to be a firewall and not a router or layer 3 switch (of the type Cisco PIX or Cisco ASA. Update 2010.10.28: It has been observed that ASA OS version 8.23 requires a static ARP entry. Use command: ”arp <interface> <Multicast IP> <Multicast MAC addr.> alias” Eksempel: arp dmz2 03bf.c0a8.831d alias ) then it is not necessary to add the static entry as these firewalls allow adding multicast addresses to the ARP table. If you're not sure weather default gateway device allows adding multicast addresses to the ARP table (meaning that it allows traffic to a specific IP address to be broadcasted to multiple ports on the switch/router), then log into the device and list the ARP table with the following command:

show ip arp

If the multicast mac address exists in the table, then you don't have to make any changes.

The configuration is done directly in Cisco’s ”privileged” mode. See screendumps below (thanks to Kim Rubeck for lending me the dumps):

A good resource on Vmware's site: Link to KB article

Thanks to Kim Rubeck for input!

EVC - CPU compability for VMotion

EVC (Enhanced VMotion Compability) increases the VMotion possibilities between different processor generations. EVC is introduced from ESX v3.5 u2 and Virtual Center v2.5 u2.

Intel Nehalem processor type is supported from ESX v3.5 u4 (and VC2.5 u2 minimum, I guess)

Link to KB article

Link to Nehalem architecture on Wikipedia

Update 2011.03.18: A BL460cG6 (E5520 Nehalem 45 nm) is compatible together with a BL460cG7 (E5620 Nehalem 32 nm Westmere) in an EVC cluster with Intel Xeon Core i7 (currently second highest level for intel processors) level enabled. However, if Intel Xeon Core i7 32 nm is chosen, then only G7 blades can be used.

Monday, April 13, 2009

Howto: Power off or kill a VM that is stuck

If you have a VM that is stuck and you can't power it off via Virtual Center, then here's some usable commands for troubleshooting and killing the VM from the Service Console:

Show running VM's:

vm-support -x
vmware-cmd -l

See the current state of a VM:

vmware-cmd /<path-to-vmx-file> getstate

Show running processes related to a given VM:

ps -auxwww grep <name-of-VM>
(Remember the 'pipe' character before grep-command - I can't display here...)

If you see an 'S' next to the process, then it can be killed. If it's a 'D', then the process is defunct and you may have to reboot the ESX server to kill the process.

Kill the proces, and thereby the VM:

kill -9 <process id>

Links to relevant threads:

VMware Communities VMTN levels

Here's the levels for VMware Communities. I always forget where to find them when I need them, so I snatched this from

Sunday, April 12, 2009

Howto: Extend system partition with Diskpart

Diskpart is a useful tool for extending logical Windows partitions (typically d-drive) which are not system partitions (c-drive) from within Windows without having to boot on a PXE-cd and rearrange partitions with e.g. Partition Magic. See here for Diskpart instructions.

Here's a workaround for extending the system or boot partition with diskpart (this will only work if the c-drive is the only partition on the .vmdk file). The logic of it is to take the whole .vmdk file and attach it to another VM as a second disk. This way Diskpart will let you extend a system partition:

1. Shut down the VM on which you whish to extend c-drive (c-drive-VM)
2. Create a new temporary VM (temp-VM)
3. Create a new disk on the temp-VM and choose 'Attach existing .vmdk file'. Browse for the .vmdk file of the c-drive-VM
4. Edit settings for the temp-VM and chose the newly added disk and resize it to new size.
5. Boot temp-VM and extend using Diskpart (you may need to assign a drive letter to disk in Disk Management)
6. Shut down temp-VM
7. Delete disk from temp-VM
8. Now the disk has been extended but it c-drive-VM it still shows as the original size. Therefore, delete the disk from c-drive-VM and add it again. This way, the correct disk size will show.
9. Boot c-drive-VM. Done.

Now, maybe this is not a super elegant way to go about it, but if you have a temp-VM ready then it's probably a little quicker than using the old way...

Saturday, April 11, 2009

Putting a host in maintenance mode hangs at 2%

When putting a host into maintenance mode, it happens that the progress hangs, or times out, at 2% and then nothing further happens. VMotion is not commenced for any of the VM's.

There are a number of things, that should be in place before automatic evacuation of VM's will work.

  • DRS should be set to "Fully automated"
  • If HA is enabled and only two hosts in the cluster, then the "Allow VM's to be powered on even if they violate availability constraints" option should be checked (see screendump below)
  • All CD's on the VM's must be disconnected. Eric Sloof from has made a cool application, vmcdconnected, where a given host can be scanned and all CD's can be disconnected with one click (see screendump below). After scan, all the VM's with a '+' in front of it has a CD-rom connected.

CPU compability tool

This tool from can be used to check the CPU compability of your physical servers. On many servers, the VT feature is disabled by default which means that 64-bit VM's cannot run. With this tool, this can be checked without having to boot into BIOS.

Draw a nice Visio digram from your VC using Powershell

If you have been drawing infrastructure diagrams in Visio of your company setup, you know how cumbersome it can be - especially as the infrastructure changes often. In stead of manually changing your Visio drawings, here's a new tool, that can depict an exact copy of your Virtual Center (now vCenter) or a given cluster - or from a given host - in Visio.

You need to have Powershell and VI Toolkit installed to run script. See here for installation instructions.

1. Go to Virtu-al for further instructions. Download the file.
2. Once extracted copy the 'My-VI-Shapes.vss' file to your 'My Documents\My Shapes' folder. If the folder does not exist create it and copy the file in.
3. Run the powershell script (Start-> VMware VI Toolkit -> VMware VI Toolkit ) with the following options:
To diagram the entire Infrastructure:
vDiagram.ps1 -VIServer MYVISERVER or HOST
To diagram a specific cluster use the following:
vDiagram.ps1 -VIServer MYVISERVER -Cluster "Production Cluster"

How to extend D-drive in Windows with Diskpart

In stead of booting into PXE-boot and starting e.g. Partition Magic for rearranging disks, Diskpart can be used for extending a logical partition which is not the system partition, typically the d-drive.

1. Shut down the VM
2. Right click the VM, choose Edit Settings, click the hard disk, type in new disk size in GB, Click OK.
3. Start VM
4. Start a CMD prompt
5. Run: Diskpart
6. Run the following commands:

List volume
Select volume 2 (if volume 2 corresponds to D drive)

7. Done. Reboot to be sure.

How to enable 64-bit in BIOS on HP server

To be able to run 64-bit VM's in VMware ESX server, then Intel-VT technology needs to be enable in BIOS. Furthermore, to enable EVC (Enhanced VMotion Compability), No-execute memory feature should be enabled, see below.

1. Go to BIOS (press F9 during boot)
2. Go to Advanced Options -> Processor Options -> Intel ® Virtualization Technology
3. Choose Enable
4. Furtermore, to enable VMware EVC, enable 'No-Execute Memory Protection' (just above Intel-VT).
5. Save and exit

NB: All hosts in your cluster should have the same BIOS settings. If not, this can result in VMotion issues.

Understanding the snapshot - how to check size of a snapshot

When creating a snapshot, the existing vmdk file is locked and a new vmdk is created, a delta file. If there are multiple vmdk's attached to the VM, seperate delta files will be created for each vmdk. If vmdks are placed on other LUNs than where the .vmx file is residing, then all delta files will be placed on the same LUN as the .vmx file. All changes made after the snapshot is taken are added to the new vmdk file(s). The delta vmdk files can grow until they reach the size of the original vmdk file. If a snapshot exists for too long, this can generate problems as the SAN LUN can run out of disk space. If this happens, the VM’s will start to crash. Therefore, as a general rule of thumb, snapshots should not be left unattended for more than one or two weeks unless it is ensured that there is sufficient space on the data store. If the snapshot is needed for a longer period, it is recommended to make a clone instead.

To check the size of the snapshot, simply browse datastore and look for a numbered vmdk file, e.g.


If a second snapshot is taken, it is named:


And so forth…

Below is a number of screenshots where you can see how files are created as snapshots are made:

1. This first sreenshot, VM is just created, no snapshot:

2. Just after first snapshot taken – no further action taken

A new vmdk file is created which is about 18 MB in size when no changes has been made yet. Remark filename, jnrrsnaphosttest-000001.vmdk

3. After installation of a couple of applications

As changes are made, the new vmdk file increases in size. In this case it increases from initial 18 MB to 198 MB. See same file as above.

4. After second snapshot taken:

When a second snapshot is created yet another vmdk file is created (e.g. server123-0000002.vmdk) and so forth...

Boot a server directly into safe mode

When doing a P2V, it is often necessary to get at VM into safe mode to disable any running HP services and afterwards unistall these HP services (in normal Windows operations). This can either be done by hitting F8 really hard upon boot, or it can be done more smoothly, as hinted by one of my colleagues:

Ones Windows is booted, Start -> Run: msconfig

Tick the /SAFEBOOT option. Reboot the machine, and it will reboot in safe mode. Uncheck this checkmark before returning to regular operations.

Submit a feature request to VMware

To submit a feature request to VMware, go to the following link:

Friday, April 10, 2009

Quick guide to installing Powershell, VI Toolkit, and Powergui

If you want to try to administer VMware infrastructure with Powershell, there are a number of applications that you need to install. Furthermore, if you want to have a GUI based interface, then Powergui is a strong tool in combination with their VMware plugin.

If it's you first time trying it out it can be a little tiring to figure out the order in which the apps are to be installed, so here's a quick getting started guide:

1. Install Microsoft Powershell. Link can be found at MS site. If you're running WinXP, look for "Windows Management Framework Core (WinRM 2.0 and Windows PowerShell 2.0)"

2. Install the VI Toolkit 1.5 (for Windows), use the same link as above. If you receive a warning related to Powershell execution policy, then openPowershell from Start -> Programs and run the following command:

Set-ExecutionPolicy remotesigned

3. Go to and install Powergui. During installation, choose to install the VI Client plugin.

4. Install the Powergui VMware Powerpack, click here. Look for the XML file, VMware.VIToolkit.powerpack. Download file. Open Powergui. Right click root note and choose import. Import the .xml you just downloaded. Done.

5. To add a host or a VC server, open Powergui, expand the VMware folder, choose 'Managed host', and choose 'Add connection' in the menu to the right. This will give you a nice recognisable view like the one from the VI client.

This site is a really cool online resource that gathers VMware related information. I especially like their vLaunchpad where you have, for exampel, a top 20 of popular blogs. Furtermore, they have a number of top ten lists with e.g. top vi admin tools wich are good to know when you're administering an enterprise setup.

Monitor progress of snapshot deletion

When deleting (comitting) a snaphot in Virtual Center, it times out after 15 minutes. To follow the progress, log into to the service console and navigate to the folder where the .vmdk files are located. Run the following command:

watch ”ls –oghut –-full-time *.vmdk”

This way, you can see when the snapshot file is removed.
This info was originally found on:

Keep track of the information flow - RSS reader

There's a lot of new information being published everyday. And it can be hard to keep track of it all. If you want to stay up to speed you have to follow quite a few web sites and blogs. To easen this task a bit, I'm using the RSS reader, Netvibes ( ). It has easy to use web interface and it gives a good overview.

VMware Communities Roundtable - Podcasts

I reasontly discovered the VMware Communities Roundtable which is a weekly recurring event (live audio broadcast) where VMware people and bloggers from the community meet and discuss different topics. The show is recorded and can be downloaded as a podcast. Duncan from and Eric Sloof from frequently join the calls.

This is a super resource if you want to be updated on what's going on and what has happened during the week. I usually hear it on the Ipod going to and from work and it's a good way to just lean back and passively receive information. However, it's more fun to hear it live, they go on Wednesdays at 21.00 h, Western European Time.

At the moment of writing, they are up to session number 43. You don't have to have an account to listen in.

Configuration of iSCSI in VMware VI3


The purpose of this post is to describe how to configure an iSCSI SAN in a VMware virtual infrastructure 3.5 with software initiator.

The prerequisites for this instruction are that the network and storage system has been configured and that you have received the following information:

ESX Hosts

  • ILO IP and credentials
  • IP address for ESX host
  • IP address for VMotion
  • FQDN for the ESX host (should be able to resolve)
  • Is ethernet traffic VLAN tagged (then you need VLAN ID) or is it only access ports?
  • Subnet, gateway, DNS servers
Storage (typically set up in closed network, 192.168.1.x/24)
  • IP addresses for the storage targets (typically 2 or 4 targets)
  • IP address for the Service Console on ESX
  • IP address for VMkernel (iSCSI) on ESX
  • Subnet and gateway
  • Make sure a LUN is made available by storage group
Read the “iSCSI Design Considerations and Deployment Guide” from VMware for detailed instructions. Just search on Google for it.

Furthermore, ensure that you have two separate NICs in the ESX host that can be used for storage. So, if it’s a Blade, then 4 NIC’s for Ethernet traffic and the two last on mezzanine card 2 for storage. The NICs can be of any type and make since the iSCSI initiator is software based and controlled by ESX on top of the NIC.

Instruction steps

0. First, below is a typical storage architecture:

1. In VI client: Make sure the ESX server is licensed for iSCSI and VMotion under Configuration -> Licensed features

2. Under Configuration -> Networking add a new virtual switch that will be used for storage. Attach the NIC’s you want to use.

3. Click properties for the new vswitch and add a Service Console 2 (COS2). Give it an ip address and subnet (typically local ip.). This second service console will receive the gateway of the first Service Console (a routable gateway ip). This is fine as it is not to be used in COS2.

4. Click properties for the new vswitch and add a VMkernel which will be used for iSCSI traffic. Label it iSCSI. Type in ip address and subnet.
After VMkernel is created enter properties for it and enter VMkernel Default gateway. This gateway ip should be the same as the IP address of COS2. So VMkernel points its gateway to the local service console.
Do not tick the box for VMotion use.

5. When done, the network configuration could like dump below:

6. Make sure the vmkernel has a gateway under “DNS and routing”

7. Go to security profile and enable software iSCSI client through the firewall:

8. Go to configuration -> storage adapters and click on the vmhba and click “properties”

9. Click Configure and then tick the “Enabled” check box and click OK.

10. On the Properties page for the software iscsi adapter, choose the Dynamic Discovery tab and enter the ip addresses of the storage targets (static targets are not supported for software initiators.)

11. Now, from the storage adapters page, rescan the HBA’s and verify that you see 2 or 4 targets (storage targets)

12. From Configuration -> Storage add the new LUN or LUN’s

13. When you have added a LUN, right click it and choose properties

For a MSA2012i with two Storage Processors (SP’s) with each to ports, there will be 4 targets (Update: In 3.5 U3 I've seen same setup but only two visible targets - but live SP fail-over works fine still). There will be 2 paths (typically on Fiber HBA’s, there are 4 because each HBA is represented with each two paths). With software initiator, there is one logical initiator and then two physical NICs teamed in the vSwitch. The initiator has two paths to two targets on the same SP.
14. Tricks:
  • Make sure that all targets can be pinged from COS2. SSH to the ESX host. From the console, SSH to COS2. From there you can ping the targets
  • If it’s a HP Blade 3000/7000 enclosure, make sure connections between the two switches used for storage are allowed (done by network department)
  • Jumbo Frames: If you are to enable it, remember to change it on all relevant parts: Storage, Network, ESX (on Switch and Port groups). Jumbo frames are not necessarily supported by the physical NIC’s. on the BL460cG1, the built-in NIC’s are supported but the HP NC326m, for example, is not. To enable jumbo frames from console, type following two commands:
    VMkernel command: esxcfg-vmknic -a -i 'ip-address vmkernel' -n 'netmask vmkernel' -m 9000 'portgroupname'
    vSwitch command: esxcfg-vswitch -m 9000 'vSwitchX'
  • Check outgoing ESX traffic: From the console, you can, when you rescan for new HBA’ and VMFS volumes, check if there is any traffic from the ESX to the targets (run command simultaneously with rescan)
  • Netstat –an grep 3260
[root@vmtris001 root]# netstat -an grep 3260
tcp 0 1 SYN_SENT
tcp 0 0 TIME_WAIT
tcp 0 1 SYN_SENT
tcp 0 0 TIME_WAIT