Virtual Infrastructure Tips - VMware

Nondisruptive upgrade of VMFS-3 to VMFS-5

2011-12-26T11:50:00.001+01:00

In vSphere 5 the VMFS filesystem has been updated to version 5 (currently 5.54). In vSphere 4.1 update 1 the VMFS version was 3.46.

In earlier versions of ESX, live upgrades of VMFS, or in-place upgrades, haven't been an option so to upgrade VMFS, basically a new LUNs had to be created and then VMs could be migrated to these new LUNs.

With vSphere 5, VMFS can be upgraded nondisruptively. This is done for each LUN by going to:

Datastore and Datastore Clusters -> Configuration -> Upgrade to VMFS-5.

It is a prerequisite that all connected hosts are running vSphere 5. The upgrade itself takes less than a minute (at least in a small test environment).

In VMFS 5, there is only one block size which is 1 MB. However, when upgrading from v3 to v5, the block size remains what it was before (see the last screendump). In the example below, the 8 MB block size is retained.

The new maximum LUN size is 64 TB - but a single .vmdk file can still not exceed 2 TB minus 512 bytes. The only way to have larger .vmdk's than 2 TB is to create an RDM and mount it as a physical device (as opposed to virtual). See this VMware whitepaper for further info.

Licensing: vSphere 5 Enterprise and 8 way VMs

2011-12-14T18:30:00.000+01:00

In my experience, more and more customers are asking for multiway VMs with more than 4 vCPUs. For my company, an IT service provider, this is a little problematic as most of our licenses are vSphere Enterprise - not Enterprise Plus.

With vSphere 5, 8 way VMs are now possible both in the Standard edition and Enterprise edition. For up to 32 way VMs, the Enterprise Plus license is required.

See link for more info, page 6.

VMXNET 3: Supported Guest Operating Systems

2011-12-07T10:11:00.000+01:00

VMXNET 3 is the newest NIC driver for VMs (requries VM HW v7). It should be chosen as default for all supported guest operating systems. Windows Server 2000, however, is not supported. Here's link to VMware KB with more info. Remember, that when you delete the old NIC and add a new one, then all IP info is wiped and should be reconfigured (mostly relevant for static IPs).

P2V with VMware Converter Standalone 5 and sync feature

2011-11-26T15:28:00.001+01:00

For this blog post number 100 (uh la laa) I decided to spice things up a bit with video tutorial showing some of the new features in VMware vCenter Converter Standalone 5 including the enhanced synchronize feature (it also existed in v4 but it didn't work too well..).

The video guides you through the migration wizard and discusses some relevant use cases for the sync feature.

Here's a link to Converter Standalone 5

Remember that as of v4.3 Windows Server 2000 is no longer supported as a source OS, so to convert win2k use Converter Standalone v4.01 in stead. In the release notes you can see supported guest operating systems.

Downloading VMware tools seperately from VMware site

2011-11-10T12:45:00.000+01:00

The other day I had to extract the VMware tools ISO for a customer. One way to do it is to log on to a VM that has not updated VMware tools to latest version, choose to upgrade VMware tools manually and then copy all of the files on the mounted ISO and make a new ISO.

However, the different versions of VMware tools can also be downloaded directly from VMware's website on http://packages.vmware.com/tools. There are also tools version for Windows servers.

I found some more info on it on this site

Got the new book by Scott Lowe - Mastering VMware vSphere 5

2011-10-20T16:11:00.000+02:00

At the last day of VMworld in Copenhagen they finally managed to get a number of copies home of Scott Lowe's new book - Mastering VMware vSphere 5 (they said they would have it by Tuesday at first). I got myself one and I'm really looking forward to reading it. One question, though, is how the book will be able to include best practices and operational experiences when the final product has 'just' come out and most of the book must have been written during the beta? But we'll see.

VMworld Europe 2012 announced - Barcelona

2011-10-20T10:52:00.000+02:00

Here at VMworld in Copenhagen, VMware has announced the location for next years 2012 VMworld Europe. It will take place in Barcelona from 16-18 October. I have to give cudos to Copehagen and the Bella Center for hosting two extremely well organised VMworlds (2010 and 2011) but I'm also looking forward to a bit warmer climate next October - providing the boss can be persuaded to buy a ticket.

How to run XenServer 6.0 on vSphere 5 - with nested Windows Server 2008 R2 VM

2011-10-11T00:41:00.002+02:00

It is possible to install XenServer 6.0 in a virtual machine on vSphere ESXi 5 and then with a few tweaks you can even run a nested Windows Server 2008 R2 VM on the virtual XenServer 6.0.

To install XenServer 6.0 in a VM, first follow this guide to configure ESXi 5.0 (or watch this youtube video).

One important step is to execute the following command from the console:

echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

Otherwise, configure like the guide. Once the custom VM has been created, to be able to choose ESXi 5 as operating system, go to Edit Settings -> Options -> Guest Operating System choose 'Other' and then choose VMware ESXi 5.x. This will ensure that you won't receive the "HVM is required for this operation" error when trying to boot the win2k8R2 vm (it is possible to change this after the install of XenServer as well).

Download the install .iso from citrix.com

Mount iso and install XenServer

When done, you will get startup screen as below

Download XenCenter from citrix.com and install

Add the the XenServer to XenCenter

Create a new VM, choose win2k8 R2 64-bit, mount ISO, install.

Done.

Upgrading vCenter v4.1 to v5.0

2011-09-08T22:12:00.000+02:00

I just upgraded my home lab vCenter server the other day from v4.1 to v5 and took some screen dumps of the installation process. The steps look fairly familiar compared to earlier versions. At one point in the installation I had an error stating that:

"The Fully Qualified Domain Name cannot be resolved. If you continue the installation, some features might not work correctly"

The reason for this error is that I had not created a reverse lookup on the DNS server. By following this guide, the issue was resolved and installation process could continue without further warnings.

Here are the screen dumps:

It was at this step that the DNS error ocurred. Below image shows how a reverse lookup zone was created on the DNS server.

Configuring iSCSI for vSphere 5

2011-09-08T21:53:00.000+02:00

Configuring a software iSCSI initiator for ESXi 5.0 is a relatively simple operation. This quick guide assumes that you have already configured an iSCSI target and published it on the network.

For inspiration, have a look at this VMware KB

Create a new vSwitch (Configuration -> Networking -> Add Networking) and add a VMkernel. Configure it with an IP address.

Go to Storage adapters and click "Add" to add a software iSCSI adapter if it does not exist already.

Once added, right click the software initiator and choose "properties".

Go to Network Configuration tab and click "Add".

Choose the vSwitch/VMkernel that you created above.

Go ot Dynamic Discovery tab and click "Add" to add an iSCSI target

You will be prompted to input IP address of the iSCSI target, just leave port 3260 as default unless you have configured it differently on your target.

Go to Configuration -> Storage and click "Add storage". Click DISK/LUN and next. If everything has been done correctly, you be able to see your published iSCSI target and can then add and format it with the new VMFS5 file system, uh lala!

vSphere 5 official release date

2011-08-17T13:21:00.004+02:00

Update: 2011.08.25: vSphere has finally been released (as of 2011.08.24)! It can be downloaded from the VMware site.

Update 2011.08.23: Well, vSphere 5 was not released yesterday as rumors would have it - so I guess we'll just have to wait and see. A guess, not completely unreasonable, would be sometime during this week or on the first day of VMworld in the US...

When: The official release date of vSphere 5 - for GA - has been set to Monday 2011.08.22.

I found info about it here and I got the same info from our license vendor.

ESXTOP to the rescue - VM latency

2011-07-21T21:49:00.010+02:00

Earlier on I have mostly used ESXTOP for basic troubleshooting reasons such as CPU ready and the like. Last weekend we had a major incident which was caused by a power outage which affected a whole server room. After the power was back on we had a number VMs that was showing very poor performance - as in it took about one hour to log in to Windows. It was quite random which VMs it was. The ESX hosts looked fine. After a bit of troubleshooting the only common denominator was that the slow VMs all resided on the same LUN. When I contacted the storage night duty the response was that there was no issue on the storage system.

I was quite sure that the issue was storage related but I needed some more data. The hosts were running v3.5 so troubleshooting towards storage is not easy.

I started ESXTOP to see if I could find some latency numbers. I found this excellent VMware KB article which pointed me in the right direction.

For VM latency, start ESXTOP and press 'v' for VM storage related performance counters.
The press 'f' to modify counters shown, then press 'h', 'i', and 'j' to toggle relevant counters (see screendump 2) - which in this case is latency stats (remember to stretch the window to see all counters)
What I found was that all affected VMs had massive latency towards the storage system for DAVG/cmd (see screendump 1) of about 700 ms (rule of thumb is that max latency should be about 20 ms). Another important counter is KAVG/cmd which is time commands spend in the VMkernel, the ESX host, (see screendump 3). So there was no latency in the ESX host and long latency towards the storage system.

After pressing the storage guys for a while, they had HP come take a look at it, and it turned out that there was a defect fiber port in the storage system. After this was replaced everything worked fine and latency went back to nearly zero.

In this case, it was only because I had proper latency data from ESXTOP that I could be almost certain that the issue was storage related.

Screendump 1

Screendump 2

Screendump 3

Changing IP and VLAN on host - no VM downtime

2011-07-17T16:44:00.005+02:00

It is possible to change the service console (COS) IP and VLAN id for hosts in a cluster without having VM downtime (see this post for changing hostname). The trick is to change the COS IP first on all hosts and then wait with the changing of the vMotion IP until all COS IP's have been changed. This way, you will be able to put the hosts into maintenance mode one by one and vMotion will still work with the old IP even though COS IP's will differ in range and VLAN id.

NB. It may be neccesary to disable HA for the cluster before you begin as the HA agent will not be able to configure on the hosts when IP's don't match for all hosts.

Enter maintenance mode
Update the DNS entry on the DNS server
Log on to the vCenter server and flush the DNS: ipconfig /flushdns
Go to ILO, DRAC or something similar for the host (you will loose remote network connection when changing the IP) and change the IP (use this KB article for inspiration): [root@server root]# esxcfg-vswif -i a.b.c.d -n w.x.y.z vswif0 , where a.b.c.d is the IP address and w.x.y.z is the subnet mask.
Change the VLAN id (in this case VLAN 12): esxcfg-vswitch -v 12 -p 'Service Console' vSwitch0
Change gateway: nano /etc/sysconfig/network
Change DNS servers: nano /etc/resolv.conf
Restart network: service network restart
Ensure that gateway can be pinged
Update the NTP server from the vSphere client if needed.
Continue the process with next host in the cluster

When all COS IP's have been changed, go to the vSphere client and change all vMotion IP addresses and VLAN id's. This will not require any downtime. And then test that vMotion works.

Done.

Changing hostname from the service console

2011-07-17T16:38:00.010+02:00

The easiest way to change the hostname is via the vSphere client (see this post for changing IP address and VLAN IP). If, however, this is not an option for some reason, the hostname can be changed from the service console the following way:

This KB article actually explains most of the proces which includes:

-----------------

1. Open the /etc/hosts file with a text editor and modify it so that it reflects the correct hostname.

2. To change the default gateway address and the hostname, edit the /etc/sysconfig/network file and change the GATEWAY and HOSTNAME parameters to the proper values.

3. For the changes to take place, reboot the host or restart the network service with the command:

[root@server root]# service network restart
Note: This command breaks any current network connections to the Service Console, but virtual machines continue to have network connection.

------------------------------

I have experienced that after a reboot, the changes are reset and the hostname is changed back to the original one. To avoid this, there is one more step to be performed (before reboot):

Change the /adv/Misc/HostName parameter in /etc/vmware/esx.conf file (see screendump)

Error during upgrade: The system call API checksum doesn’t match

2011-06-28T09:04:00.003+02:00

Today, I got an error during upgrade from vSphere 4.0 to 4.1 stating something like:

The system call API checksum doesn’t match

There was a lot of similar lines filling the console. I was a bit worried that the upgrade had gone wrong even though I had done three similar upgrades before this one with no errors - and that I would have to reinstall in stead.

Luckily, I found this error description in the 4.1 release notes stating that a reboot will fix the issue. So I waited for a while to be sure that the upgrade finished, rebooted, and everything looks fine:

Link to release notes:

"ESX service console displays error messages when upgrading from ESX 4.0 or ESX 4.1 to ESX 4.1 Update 1

When you upgrade from ESX 4.0 or ESX 4.1 release to ESX 4.1 Update 1, the service console might display error messages similar to the following:
On the ESX 4.0 host: Error during version check: The system call API checksum doesn’t match"
On the ESX 4.1 host: Vmkctl & VMkernel Mismatch,Signature mismatch between Vmkctl & Vmkernel

You can ignore the messages.

Workaround: Reboot the ESX 4.1 Update 1 host. "

Used network ports between ESX, vCenter, and the vSphere client

2011-06-21T08:39:00.007+02:00

In a single customer setup, the network traffic and ports used in the virtual infrastructure are typically not a focus area because most components can be placed on the same networks. However, in a multiple customer environment we're experiencing that the network guys are asking a lot of questions as they want to lock down and secure the networks (which makes sense..).

Anyway, I just wanted to gather some of the info that I use regularly.

A link to my previous post with a network diagram
Link to VMware KB article about used ports for a vSphere environment and related components
vMotion between networks requires TCP port 8000
If searching and sorting VMs in the vSphere client is slow, then ensure that port 8443 TCP is opened between the vSphere client and the ESX hosts
If you can't get a remote console on the VMs (you get a black screen and a yellow bar in the top stating some sort of MKS error) ensure that port 903 TCP is allowed between the vSphere client and the ESX hosts
If an ESX host keeps disconnecting in vCenter, ensure that port 902 UDP is allowed from the ESX host to the vCenter

Required ports between ESX and vCenter:

Source	Destination	Direction	Protocol	Port	Purpose
vCenter	ESX	In/out	TCP	902	VMware console
vCenter	ESX	In/out	TCP	903	VMware console
vCenter	ESX	In/out	TCP	443	HTTPS
vCenter	ESX	In/out	TCP	22	SSH
vCenter	ESX	In/out	TCP	80	HTTP
vCenter	ESX	In/out	TCP	161	SNMP
vCenter	ESX	In/out	TCP	5989	CIM
ESX	vCenter	out	UDP	902	Heartbeart

vSphere Network Ports Diagram

2011-05-10T12:37:00.004+02:00

Here's a great network diagram showing used ports in a vSphere environment made by Dudley Smith. Go here for current version as of 2011.05.10. I had to look for it a bit so thought I'd post the link.

And also, there's the vSphere vReference which is quite handy

Installing View 4.6 in home lab

2011-04-18T22:52:00.008+02:00

After recently finishing my home lab ESXi 4.1 installation (the Blue Bad Boy) I thought I'd put it to good use. I decided to do a full View 4.6 installation with external access over PCoIP via a Security Gateway. After getting it all up and running, I must admit that it was a bit more work than initially expected - but it has been a lot of fun.

In this post I will not go into detailed installation steps, in stead I'll try and give an overview of the components that I have used (local mode and linked clones not included) and then link to the posts I've used for inspiration.

Components

First of all, a vCenter installation and a domain controller are required. I have chosen to go with Windows Server 2008 R2 but other than that it is pretty much standard installations.

The main component of the View installation is the Connection Server. And then there is the Security Server which is basically a subset of features from the Connection Server. After installation it is linked to the Connection Server from the Connection Server administrative web interface - and it is also configured from there.

I used this excellent guide by Poul Slager to install the Connection Server. I did the same as Poul and installed just one Win7 VM with the View agent on it and added it to a static pool.

A new feature in View 4.6 is that the PCoIP protocol can now be used also from external sources (e.g. from outside the company network) but this requires a Security Server. The Security Server is typically placed in a DMZ and it is the Security Server which establishes the PCoIP connection directly to virtual desktop.

At the VMware View blog, there's a post with a 40 minute video explaining the infrastructure and new features of View 4.6.

For the specific configurations for enabling PCoIP from external sources, I used the Setting up PCoIP Remote Access with View 4.6 document.

I experienced a strange error when at first I connected to the Security Server from and external source. It worked fine internally but from the outside I could connect and authenticate but then the remote connection just showed a black screen for about 10 seconds and the connection closed. In the View desktop event viewer there was en entry stating: "Closed PCoIP connection doesn't match global value". To fix this I adjusted the configuration in the Connection Server under View Configuration -> Servers and made sure that the external URLs for the Security Server and the Connection Server were identical. The external URL was set for the actual outside URL in both cases and the IP was set for the outside ip of the ADSL modem in both cases - this solved the issue in my case (see screen dumps below).

Currently, with all the components running, the setup is taking up about 10 GB of memory, so there's still room to load up the ESXi box, it has a total of 16 GB, with more VMs! (see screendump below).

Networking

For routing and firewall internally between the infrastructure components I chose a Vyatta virtual appliance which I downloaded from VMware Marketplace. Per default, this appliance included three NICs which suited my requirements for creating an inside LAN, outside LAN, and a DMZ for the security server. On the vSwitch I have created three different VM networks. However, I have not VLAN tagged any of the networks as only one ip range will leave the physical ports on the switch (the Vyatta router acts as gateway for all the infrastructure components).

The learning curve for the Vyatta is quite steep in my opnion. I have spend my fair share of hours trying to figure out the logic of the NAT, DNAT, and the firewal rules. For configuration I have been using a mix between the web gui and the CLI. The CLI is actually quite nice when you get used to it (TAB is your friend).

Remember to save your configurations to disk before rebooting or you will loose all configurations (I learned this a couple of times ;-)). So obviously type 'configure' to into configuration mode and then 'commit' when your done. 'Exit' to exit configuration mode. 'save config.boot' to save configuration to disk. Default credentials for the vyatta is user: vyatta, pw: vyatta.

To get started and setup the Vyatta I used the Quick Start Guide which you can get at vyatta.org. At the site there is also a quick start video which is useful.

And then for firewall configuration etc. I used this guide which worked surprisingly well.

The basic principle for the router in this setup is that you want to allow all traffic from the Inside Lan and the DMZ to be able to get out to the internet. You also want your Inside LAN to be able to access the DMZ. All traffic from the Outside entering the gateway NIC on the router should be dropped. However from all addresses on the Internet, access on port 4172 should be allowed (and directed) only to the security server. And then only the Security server's IP will be allowed to open connections on the same port to the inside LAN. So for 'opening up' a port in the firewall you will need both a firewall rule and a DNAT rule (destination NAT). This last part had me quite confused.

So, the final setup currently configured according to the diagram below. They way I used is to connect to the View Desktop and from there I can open a vSphere client and have full access to the vSphere home lab.

My ESXi home lab - the Blue Bad Boy

2011-04-17T15:36:00.010+02:00

A while back, I decided to build my own home lab whitebox (the Blue Bad Boy) with ESXi 4.1 U1. I've been running Workstation on my laptop with 4 GB memory for some years but the limitations to this setup is obvious. At work we do have a number of test servers that you can play around with but you still have to be a bit more careful than you would in a home setup.

Once the decision was taken, about a million questions followed. I wanted a setup that was similar to our production environment and that could do all the enterprise features such as HA, vMotion, FT, etc. Furthermore, there should be sufficient capacity to run a View 4.6 installation and a vCloud director setup which both require a number of infrastructure components.

So should it be one or two physical servers and what about a NAS box? The full blown setup, it turned out, would be way to expensive for my budget. So I decided to go with one physical box and then with an option to expand with a NAS box later on. For vMotion etc., this could be done with two virtual ESXi's and nested VMs.

There are quite a number of good blog posts and web sites about building home labs. I was leaning towards replicating the BabyDragon setup but two things kept me back. 1) The motherboard was about double the price in Denmark (if you buy from the States they will slaughter you with extra VAT and import taxes) and 2) There's already a number of people who have done this setup so it just seemed a bit too easy.

I ended up leaning towards a setup posted by VMwArune which included a real nice Intel Server Motherboard with dual port GigE ethernet.

Hardware parts

Motherboard

The motherboard is an Intel Server Board S3420GPV which is on the HCL. Form factor is ATX and it sports an integrated dual-port intel NIC (also an the HCL) - so it is not necessary to inject custom drivers or to buy additional Intel NICs (which are relatively expensive). Up to six SATA disks, no SAS. Max 16 GB unbuffered ECC memory. Socket 1156. One internal USB port for ESXi dongle. Unfortunately, it does not have KVM over IP as the Supermicro X8SILF board has.

CPU

For the CPU, I chose the Intel X3440 (on the HCL) which is a 2,53 Ghz quad core processor with hyperthreading. The X3430 was somewhat cheaper but did not have hyperthreading and the X3450 was a bit more expensive but the only difference was the clock frequency (I'm not totally sure it will support FT, though...)

Memory

16 GB (4 x 4) of unbuffered ECC memory, DDR3 (KVR1333D3E9S/4G). The motherboard only supports the more expensive ECC server memory (registered or unbuffered ECC) so that was a bit of a draw back. However, I did run it for a couple of days with regular non-parity non-ECC desktop memory and it worked fine.

Hard drive

I really wanted to get an SSD disk with 128 GB and then a 7200 RPM spindle with more capacity. But the SSD's are quite expensive and as I'm maybe going for NAS later I did not want to spend too much on storage up front. I decided to go with a Samsung F3 1 TB 7200 RPM.

USB dongle

1 x 4 GB regular Kingston DataTraveler for installing ESXi on.

Power supply

From what I understand, these whitebox home labs do not require that much power. So I chose a 430 watt Corsair CX power supply. Not much to say about that.

Chassis

For the chassis I chose a Cooler Master 430 Elite Black. I guess it could be any ATX compatible chassis, but this one was not too big and it is very affordable - and it has a nice glass pane on the side. After I bought it I saw that there's even smaller ATX chassis, the Elite 360, but it only has room for one or two disks.

Ethernet Switch

I wanted a VLAN tag enabled and manageable GigE switch. The HP Procurve 1810G series (8 ports) switch seemed to deliver just that - and again - affordable.

Pimping

Just to spice it up a bit - and because the chassis already holds a blue LED 120 mm fan, I have installed a Revoltec Kold Katode twin set (blue light..).

Inital experiences

I had to go through somewhat of a troubleshooting phase before I had ESXi 4.1 update 1 properly up and running. I was experiencing some very strange errors during install as I couldn't get passed the Welcome screen. If I tried ESX classic (v4.1, v4.0) it would hang in different places while loading drivers. So I updated the BIOS and that didn't help. I tried unplugging USB devices (the CD-ROM is external). Then I found out that the main board only supports ECC memory and I had bought non-ECC memory. So I was pretty sure that the memory was the fault. But - as I returned the memory, I bought a new cheap USB keyboard as I had seen some posts where people had USB keyboard issues. And low and behold - as soon as I changed the keyboard (I was using a Logitech G510 gaming keyboard to begin with), the installation went through clean. And that was even with 4 GB of non-ECC DDR3 memory from my other desktop.

Anyway, the beast is now up and running and everything works like a charm. And it's very quiet. I'd seen posts from ultimo 2010 where people couldn't get the second NIC to work - but it's been working fine for me.

Price

I've ordered all the parts in Denmark but I'll convert prices to Euro so it makes more sense. The total for the whole setup including the HP switch is about 925 EUR (~ 1.332 USD) so it's actually not that bad.

1TB Samsung 7200rpm 32MB SATA2 58 EUR

Intel Server Board S3420GPV - ATX - Intel 3420 160 EUR

INTEL XEON X3440 2530MHz 8MB LGA1156 BOX 208 EUR

Memory 16 GB Kingston unbuffered ECC 253 EUR

Cooler Master Elite 430 Black no/PSU 49 EUR

430W Corsair CX CMPSU-430CXEU 120mm 47 EUR

KINGSTON DataTraveler I 4GB Gen2 Yellow 17 EUR

HP ProCurve 1810G-8 Switch 97 EUR

Twin katode lights 10 EUR

Shipping ~ 27 EUR

Total 925 EUR

Capacity Planner - reports don't work in 2.8.0

2011-02-10T18:35:00.004+01:00

After the recent upgrade of Capacity Planner to version 2.8.0 a number of minor changes have been introduced some of which has been mentioned in a previous post. But there is one change in particular which has been giving me grey hair.

The problem was the Reports link did not work: The first project I created, after the upgrade, was done in the regular fashion using the wizard, choosing CA or CE etc. After this I registered the Collector database ID and started uploading data. Data was sent fine. But it did not show up in the Dashboard even after a week of waiting. In the projects overview it stated for the project that there was 'not enough data' and the collector success rate was at 0%. Clicking on the project gave the following info: 'Cannot display dashboard: no statistics data has been collected'.

But there are two tabs on the Dashboard front page of which the default is Projects and the other one is Companies. My data was showing under the Companies tab, not under Projects, as I had not created a separate project for this customer/company.

And here's the part that was driving me nuts. Even though I could see the data, I could not run the reports. When I tried clicking the reports link nothing happened - there was no drop down box and mouse-over on the link showed 'javascript:void(0)'. This is quite a big deal as I'm usually using the Assessment reports as the final product of the Capacity Planner for the customer.

The trick to fix it was to create a new project and attach it to the existing customer (this can be done during the wizard - after the customer has been created). And then, also as part of the wizard, you can choose the relevant servers to be included in that particular project (I chose all). And then you have to wait a one or two days for the data warehouse to process the changes (yes it can take a couple of days...). After this, the Reports -> Optimization Reports link will be available again.

Third post in the Top 5 Planet V12N

2011-02-10T18:02:00.003+01:00

More or less by coincidence I discovered that one of my posts had made it to the weekly V12N top 5 blog posts at the VMTN blog - which to me is quite an honour. It means, I guess, that someone else finds your writings useful to some extent.

This is the third post I have on the Top 5 list:

Week 5, 2011

Capacity Planner 2.8.0 released - A new version of VMware Capacity Planner v2.8.0 build 46460 has been released (only available to partners). The most notable new features are support for desktop assessments and that multiple assessments can be grouped under one company. The collector interface is more or less unchanged.

Week 51, 2010

P2V pre-migration checklist - and considerations - My previous post was a P2V post migration checklist. This post is a pre-migration checklist which is about all the information that should gathered and checked before doing any P2V conversions. I have been involved in a number of larger P2V projects (+50 P2V's) and, in my experience, proper planning is a key element for a successful project. Typically, you, as a VMware- or P2V person, have no real knowledge of the Windows servers to be converted - their just another server.

Week 48, 2009

VLAN testing in ESX 3.5 - The traditional way of testing is to create a vSwitch with only one vmnic connected. Then connect a VM on that vSwitch with one of the VLANs. Configure an IP address in the address space of the VLAN and ping the gateway. Do this for all the VLANs, and then connect the next vmnic to the vSwitch and start over. The following method speeds up VLAN testing significantly (in this case from 100 to 16 test cases). It is not totally automated, but I have found it very useful nonetheless.

Capacity Planner - uploading to Dashboard without internet access

2011-02-08T21:39:00.004+01:00

Over the last couple of weeks, I've been setting up a Capacity Planner at a customer site (not full time, though). There have been the usual complications of not receiving user accounts with the proper access rights but also this time there was trouble uploading data from the Collector to the Dashboard. The customer uses a proxy server and even though we had specified as a requirement that TCP port 80 and 443 outbound should be opened, they weren't. I could register the collector but no files were transferred to the Dashboard and files stayed in the 'outbox'. After a while they found another proxy that could be used, and right away after reconfiguring the proxy from within the collector, I could see files started transferring (It's quite nice to see when the tool actually works as expected - because it doesn't always in my opinion...)

So I was looking around for another way to upload the files and it turns out, that this can be done fairly easily. The process is fairly straight forward. On the primary collector you generate the files to be uploaded but then you leave them in the outbox. Then you copy the files from the outbox and put them in the outbox of another collector that works and has proper internet access. This secondary collector does not need to registered to the specific project or customer as the files have already been tagged with an ID and the Dashboard will know where to put the files. I followed this KB article to move files to another collector.

Another learning point was that if you're unsure that your files are actually uploaded all the way to the Dashboard and you want have a backup, then enable the 'archiving' function on the collector. That way you will keep all info and will be able to re-upload at a later stage. See this KB article for reference. Make sure to have enough disk space available. After one week of collecting data without sending any to the Dashboard for 150 servers - the log files amounted to about 1 GB of data (for a full CA analysis).

And bonus info: After the update to 2.8.0, they have change Dashboard somewhat. Now, data can end up both under a given 'project', which is the default view when you login, or it can end up under the 'customer' tab (there are two different tabs in your customer overview). This, to me, was quite confusing as I was expecting to see data arriving under the 'projects' tab which was reporting 'no data collected' for the customer even though I had been uploading data for almost one week.

ESX 4.1 install error on BL460c G7 - NIC driver fails to load

2011-02-08T16:11:00.010+01:00

The HP BL460c G7 is on the VMware HCL list for ESX 4.1. However, when trying to install ESX 4.1 there's an error during install - it fails to load drivers for the network adapter ("No network adapters were detected"). It doesn't help to update all firmware to latest version (even though this should be done in any case...) (Update 2011.02.18: This problem persists on ESX 4.1 U1)

This is a known error and there's a fix for it. However, it seems strange that the G7 blade has made it to the HCL list...

The problem is that the Integrated NC553i Dual Port FlexFabric 10Gb NIC driver is not included in VMware's installation ISO for ESX 4.1. There are two ways to solve the issue. One is to load a custom set of drivers for the NIC and the other is to use an HP VMware install image. If you're using ESXi or scripted installation of ESX classic, then you have to use the HP image.

(Update 2011.07.19: Custom HBA driver should also be loaded during installation - simply load both ISOs)

Custom NIC driver from VMware can be downloaded here.

Custom HBA driver from VMware can be downloaded here.

(Update 2011.02.18: Apparently, the NIC drivers are updated quite frequently at the moment. Go to this main link and then 'plus out' Driver CDs to find the most recent one.)

HP image for ESX(i) can be downloaded from here.

The custom driver, when downloaded, is in an ISO format. To load it during installation, do the following:

Upload the ISO file to where you have the ESX installation image
On the Custom Drivers page in the wizard, choose Yes and click on Add. It will tell you to load the driver CD (see picture below)
Unmount the ESX installation CD and mount the driver ISO in stead. This can be done without interrupting the installation. You will be prompted to verify the custom driver package, click OK (see picture below)
That's it. at a later stage in the wizard, you will be prompted to reinsert the installation ISO. Do that when prompted.

Capacity Planner 2.8.0 released

2011-02-02T10:52:00.005+01:00

A new version of VMware Capacity Planner v2.8.0 build 46460 has been released (only available to partners). The most notable new features are support for desktop assesments and that multiple assesments can be grouped under one company. The collector interface is more or less unchanged.

Unfortunately, it doesn't look like VMware is spending too many resources developing this tool.

More importantly, support for Windows 2008 server for the collector has been added (see installation guide page 8, requires login)

Below is the "what's new" part of the release notes:

WHAT'S NEW IN CAPACITY PLANNER 2.8

The following items are the main changes in this version.

· Desktop Assessments. In addition to CA and CE assessments, users can create Desktop Assessments. These assessments facilitate analyzing a Desktop environment for virtualization. To support Desktop assessments, additional objects are collected for analysis.

· Collector or Dashboard Inventory Additions. Information about Users, Login Profiles, Logon Sessions, User Groups and Networks is collected in addition to the existing objects in CP 2.7.x. The Dashboard provides different views that help users identify the Users, Locations, Networks and Desktops that are View Ready.

· Separate Projects and Companies. Instead of creating a separate company for each type of Assessment, users can create multiple CA, CE, or Desktop projects within the same company. They can use the same data collected within a company to perform different Assessments. The landing page is redesigned to show separate Projects and Companies tab.

· Hardware Profiles. Hardware Profiles allow tags to describe the Hardware. Hardware Profiles can represent different types of Hardware such as CPU, Chassis, Adaptor NIC etc. Users can create Hardware Profiles representing specific devices and have the option to mark them View Ready. This information is used in identifying whether Desktops having the corresponding hardware are View ready.

· Reporting Enhancement. Optimization reports have been modified to include View Readiness information for Users, Networks, Locations, and Desktops.

· Templates Update. Cluster and Shared Storage templates are added to specify Cluster and Shared Storage configuration. These templates can be included in Scenarios for placement of systems.

· Scenario Improvement. The scenario can now be marked for Desktops or Servers. Users have the Cross Boundaries option for Networks. The Scenarios can now build clusters considering shared storage.

VMworld 2011 in Copenhagen - again!

2011-01-29T21:52:00.004+01:00

The locations of this years VMworld 2011 have been announced at the vmworld.com site. It will be in Las Vegas (like in 2008) from August 29-September 1 and in Copenhagen, and I assume it will be in the Bella Center like last year, from October 18-20.

I did hear some rumors about Barcelona for the European conference but I must admit that, even though it's not as exotic, the conference in Copenhagen was very well organised with nice facilities and easy transportation.

Online backup for personal computer

2010-12-11T10:26:00.004+01:00

For a while now I have been trying to persuade myself to start using an online backup service for my private documents, pictures, emails etc. The thing is that over the years I have, like most others I guess, built up quite a lot of personal data which it would be impossible to recreate should it be lost.

Until now I have backed up data to from the laptop to an external USB drive. This method works smoothly with the built-in backup software on the USB drive from Seagate. And I will be able to restore data should the disk fail or if I lost the laptop. However, for the most part the external disk with the backup data was, and is, lying just next to laptop. So in case of a break in, a thief would most likely take both the laptop and the usb disk. Or in case of a fire or other such extreme cases both laptop and backup will be lost. The risk is small but the impact, from my personal perspective, is rather high.

After looking online backup I found the following things which was reassuring:

Online backup seems to be a relatively mature market. There are quite a few players, and prices seem competitive. A one year subscription ranges from 50-100 US$.

Security seems to be in adequate . Most providers encrypt data before, while, and after it has been uploaded. Unfortunately, encryption is only protected by password (at least for the solution that I chose) so it's not this double security approach where it's something you have (like a RSA token or PKI) and something you know (the password). However, for my chosen solution, I first have to log in, and then I have to provide a strong password, which is not stored at the service provider, to acutally restore data.

Of the bigger players can be mentioned IDrive, Mozy, Carbonite, Crashplan, and BackBlaze. I read a few reviews and it seems that, a part for some minor differences, they offer pretty much the same service, so one should be pretty safe to choose either one of them.

Here's link to two reviews: From Notebook Review and from Digital Inspiration

I was leaning towards Crashplan mostly because the interface appealed to me and a colleague recommended it. However, I ended up choosing IDrive for the following reasons:

It's been around for longer than the other players (you don't want your backup provider to go out of business).
Phone support (when trouble hits it's always nice to be able to call someone...)
Price was right for my needs (50 US$/year for one computer and 150 GB storage)
Online browse and restore of files
Continuous backup and file versioning
Sufficient security
Status reports via email

My experience after a weeks use is that maybe it is not the fanciest of interfaces, b

ut it gets the job done. After three or four days I had uploaded my ~25 GB of data over the ADSL connection and I have tested restore of files from the web interface successfully. And email notifications work.

On a final note I can mention that I have been using Dropbox for a while to be able to share files between computers and to be able to reach files online. It's limited to 2 GB in the free edition but it can be highly recommended. It supplements the backup application but it cannot substitute it in my opinion.

Installing a web server on an Amazon AWS free VM

2010-11-23T18:23:00.005+01:00

In the previous post I described how you get a free linux VM in the Amazon AWS cloud up and running. This post will describe how you can use it for something practical.

Apt-get is not installed on this Micro Instance VM. So at first I tried to do a manual install of Apache by simply uploading the .tar.gz files to the VM via WinSCP and tried to run the .configure file. This didn't work as a C compiler was not installed on the system. I went on to look for GCC and got that installed and then I could install Apache. For some reason it didn't quite work, though. And also it's mayby a little too much work to get a web server up and running...

Then I stumbled upon the Yum command which is similar to Apt-get and which is actually pre-installed in the VM and is working out of the box.

With Yum, installation is a breeze. Issue the following commands:

#sudo yum install httpd
#sudo chkconfig httpd on
#sudo /etc/init.d/httpd start

The sudo command will not prompt you for a password but will let you execute commands as root. You can't su -root... (alternatively, you can try sudo -i to get a root shell)

If it complains about a missing C compiler, then install it this way:

#sudo yum install gcc

The web server installs its .conf file in /etc/httpd/conf/httpd.conf. There's is the usual test page displayed until you place an index.html file in the /var/www/html folder.

Free linux cloud VM with Amazon Web Services (AWS)

2010-11-23T17:27:00.010+01:00

Recently, Amazon announced that you can get a free linux VM for one year in their public cloud solution - Amazon Web Services (AWS). They call it a Micro Instance and it's got something like 1 vCPU, 600 MB of memory, and 10 GB storage, see specs here. You get full access to the VM via SSH but there's no console access as such.

So I decided to give it a try.

First, you need to create an AWS account (there's a link on the front page..). They need a valid creditcard for that. Then you log into the AWS Management Console. This requires you to register again. They had implemented a rather odd security feature where they call your mobile phone and you have to punch in a pin-number to confirm. I must admit that, for testing purposes, this wasn't the most smooth registration process.

Once into the AWS Management Console you're presented with a number of tabs. The first one is Amazon S3 which is an online file placeholder (i guess like an FTP server). To create your VM, go to the Amazon EC2 tab and click on Launch Instance (see below). This process is fairly simple. It is not quite easy, though, to see exactly which one is the free edition, but I just chose the minimum specs available to be on the safe side. And look for something like linux and Micro Instance.

Firewall rules are easy to configure via the web interface. You can add some pre-defined ports such as mail, web, etc. Port 22 is enabled by default.

A KPI keyset is generated (for authentication purposes) and you can download the .pem file to your local harddrive. They give an example of howto login via ssh from a console and use the generated key. Example:

ssh -i keyname.pem root@vmname.eu-west-1.compute.amazonaws.com

If you use this command will receive a login error as root cannot login directly. So just change 'root' in front of the @ with the, in the error message, suggested 'ec2-user'.

Once logged in you can execute commands as root with the 'sudo' command. It will not prompt for a password. Or alternatively use sudo -i to get a root console. But you can't su - root.

If you want to use Putty to acces the VM directly, then you have to convert the .pem file to a .ppk file. This is easily done using this guide.

To use the .ppk file, open Putty and go to SSH -> Auth and browse to the directory where you stored the file. And then you connect to the VM (saving the profile will save you some time at next login..). There's no password.

The same .ppk file can also be used for WinSCP which is handy for uploading files directly to the VM.

As you have a public DNS name, this can be used to create an eiasier to remember C-name DNS that you can point to the generated machine name.

So far so good. Now there's access via SSH. Then I tried to configure a simple web server. I'll describe that in the next post.

vMotion between firewalls

2010-11-17T15:44:00.002+01:00

Currently, I'm setting up a new VMware cluster as the exsiting hardware needs to be retired. The new cluster is in another management zone (and in another vCenter). To minimise downtime I looked at doing vMotion between the two clusters.

What I did was to disconnect one host from vCenter. Then add the host to the other vCenter directly on the ip number. The host was not added to the newly created cluster, only to the datacenter. And then drag and drop VMs between the clusters (EVC was enabled).

There was a couple of things that had to be tweaked before it worked.

vMotion had to be done between firewalls. When doing this, there are two important things to remember:

1. Set the default gatway of the vMotion interface (via vSphere Client)

2. Open inbound/outbound on port 8000 TCP in the firewall (see ESX configuration guide, page 150).

Furthermore, I encountered another issue. A number of VMs had a vmxnet NIC (it's some old VMs...). When starting to vMotion there was a warning that vmxnet is not supported on target host which is ESX 4 (source was ESX 3.5). However, after vMotion, the vmxnet NIC still worked. I tried to update VMware Tools and virtual hardware version to v7 and that also worked. vmxnet is kept as NIC after upgrade.

Hands-on labs - that I did...

2010-10-13T16:45:00.008+02:00

This post is mostly for myself to keep track of the labs that i did at VMworld 2010 and what struck me as useful features...

vSphere 4.1 - new features

Storage IO shares and limits. They're not quite there yet with Storage DRS but this is a first step.

DRS feature - possibility to keep one or more VMs bound to a specific host. This is practical in relation to licensing issues where you pay per physical core/socket in a cluster.

8 concurrent VMotions!

Better HA info with new Cluster info link. It will aggregate relevant HA info and display it in a window.

Scripting ESXi installation with PXE boot and get the files over the network from a tftp server looked pretty straight forward.

VMware View 4.5 - install and configure

Components: Main View Manager server (not the correct name, I believe..). Transfer server for offline clients. View agent installs on the VM. View client to connect to View Manager.

vCloud Director - install and configure

Basically a VM that install on top of your VI and communicates with vCenter. Works as administration unit and self service portal to customers/admins. Installs on a RHEL 5 U4 or up, 64-bit. Licensing is done per VM. vShield Manager is required.

Update Manager

Practical for host patching of course... Also relevant for upgrading VMware Tools and HW version. Tools and HW version can be done in a single script.

Thinapp 4.6 - new features

The new feature that IE6 can be packaged easily is very useful. If you package an app on WinXP it can be run on other OS'es, e.g. Win7, without modification. Every time I play around with Thinapp it always strikes me why we don't use it more...

PowerCLI 4.1

One of the first examples is a script to go through your VMs and find and delete all snapshots older than a given date - e.g. older than seven days. This example is spot on and something that we're working on imlementing as well.

vShield Zones

One thing that I noted in this lab is that vShield has a built in load balancer which was easy to configure. This could probably be a good substitute for MS NLB.

VMworld 2010 - Hands-on labs, first impressions

2010-10-12T14:43:00.007+02:00

There's been a lot of talk about the hands-on labs at this years VMworld. Here in Copenhagen there are around 240 thin clients with each two screens running up against two datacenters in Florida and Virginia with a dedicated 100 Mbit line over the Atlantic. There's also a fail-over possibility to a datacenter in Europe, should an issue occur. The whole solution is based on a number of Lab Manager installations with a custom interface on top of it which has been built for the purpose.
So far, I have taken four labs, and I must admit that I'm impressed. It works so well that you're not even thinking about what is going on under the hood. What you're actually experiencing is that as soon as you're choosing a lab, then almost instantly you have two ESX servers and a number of VMs provisioned for you. A VMware employee told me that part of the custom code is calculating the most popular labs, and then pre-deploying a number of these up front as to reduce waiting time.
Furthermore, the whole user experience is pretty cool. They've made a GUI for choosing the labs, and on flatscreens around in the room you can see statistics such as most popular labs and total number of VMs created and labs completed (see pictures below).

Restart of ESX management agents

2010-09-28T14:15:00.002+02:00

This is just a post to remember the commands for restarting the management agents on an VMware ESX server:

#service mgmt-vmware restart

#service vmware-vpxa restart (the HA agent)

Both of these agents can be restarted without affecting VM operation. Restarting them can be a useful step in troubleshooting if vCenter has trouble connecting to a host or if you experience HA errors.

Here's link to KB article

For restarting mgmt agents in ESXi, this can be done via the console menu interface, see link above.

Console-setup - service console tool for network config in ESX4

2010-09-24T16:08:00.004+02:00

As of ESX 4.0 U2 a new tool for configuring network in the service console (COS) has been introduced. If you're not too comfortable with CLI then this might come in handy. The tool will give you a numbered menu and you can list and configure nics, vswitches, vswifs, etc.

Here's a link to a VMware KB article that presents the tool.

To run the tool, type console-setup in the COS.

Menu entry 1, 2, and 3 will show the output of esxcfg-vswif -l, esxcfg-nics -l, and esxcfg-vswitch -l respectively.

Menu entry 5 will let you configure your service console without having to remember any of the commands. Pretty neat..

Here's a link on how to do it the old school way.

What does ESX stand for?

2010-08-03T08:43:00.002+02:00

Not too long ago I was doing some general VMware introduction to a number of colleauges from our Chinese branch. One of the guys asked what ESX was short for. And I had actually no idea. It's a product I've been working with for several years and yet it hadn't ocurred to that ESX was probably more than just a three letter name.

Today, I saw an article on acronyms on Yellow-Bricks which reveals the secret. Furthermore, there's a link to a video interview with Mike DiPetrillo which elaborates on the matter (about 20 minutes in...)

So:

ESX: Elastic Sky X

GSX: Ground Storm X

The two names was invented by some marketing people hired by VMware. VMware didn't like it too much so they shortened it to ES and GS and the X was added just to make it sound more technical!

Disaster recovery: Procedure in case of site failure

2010-07-08T15:22:00.007+02:00

Here's a short example of a procedure for recovering a VMware cluster from a site failure. The example scenario consists of two ESX4 hosts on replicated storage divided on seperate locations. There's no automatic failover for storage between sites, manual breaking of the mirror is required.

1.	Log into vCenter and verify whether or not storage is available for the cluster. If storage is unavailable, create an incident ticket for the storage group with priority urgent and with a request to: “Manually break the mirror for the “"Customer X" replicated storage group” used by ESXA and ESXB” The ticket should be followed by a phone call to the storage day/night duty to notify of the situation.
2.	When mirror has been broken, rescan remaining hosts in the cluster. This rescan can possibly time out. If this happens, reboot the hosts. After the rescan/reboot all shared LUNs will be missing on the hosts. These should be added/mounted manually from the console (step 3) (in ESX4u1 there's a bug in the add "storage" wizzard, so it doesn't work from the vSphere client, see this post for more info)
3.	Putty to each of the hosts and run the following commands: #esxcfg-volume –l This will list available volumes. For each volume, run the following command: #esxcfg-volume –M label or UUID> For example: #esxcfg-volume –M PSAM_REPL_001 See screendump below for further exemplification:
4.	From the vSphere client, for each of the available hosts go to Configuration -> Storage and click “Refresh”. Verify that all LUNs appear as before the site failure
5.	Power on all VMs
6.	Done. In this situation, storage will run from the secondary site. The storage group will be able to reverse the replication seamlessly at a later stage when failed site is operational again. This does not require involvement from the VMware group.

Site redundancy with manual breaking of storage mirror

2010-07-08T14:48:00.004+02:00

We have just installed a site redundant cluster for a customer. The cluster consists of two ESX4 hosts on replicated EMC CLariion storage. The ESX servers as well as the storage reside on different locations (preferrably we would have liked to have done it with storage virtualisation and seamless storage failover ala Datacore or SVC, but this was not an option..).

The site redundancy is enabled by using replicated storage. Should the site with the active LUNs fail, then the storage mirror can be broken manually and operation can be resumed on the remaining site.

One thing we discovered we that resignaturing of the LUNs is no longer necessary, as it was in previous versions when a mirror had been broken. This means that LUNs can be remounted directly without modifications, see Fibre Channel SAN Configuration guide pp. 74-76.

Earlier, you had to first break the mirror, then resignature your LUNs with the advanced feature LVM.resignature and then add the LUNs. This changed the UUID (and the label on the LUNs for that matter) which means that all VM had to be manually reregistered in virtualcenter. This is a bit time consuming and not something you want to spend yor time on in a disaster scenario.

In vCenter, you can use the "add storage" wizzard to remount the LUNs. However, there's a known bug in the software so it does not work. In stead, it has has to be done from command line with the following command (rescan the HBAs first. if it hangs, then reboot):

# esxcfg-volume -l (to list available volumes)

# esxcfg-volume -M (to persistently mount volume)

See this post for example site recovery procedure

My VMworld session ready for public voting

2010-05-20T10:43:00.006+02:00

Update: Unfortunately, my session was not among the lucky winners. Apparantly, the world is not ready for exciting service descriptions ;-) In stead, I'll be going to VMworld in CPH as an attendee.

My session has passed the internal review and is now ready for public voting. It is placed under 'Private Cloud - Management' and the title is:

Defining your services and offerings on vSphere

Description:

As virtual infrastructures (VI) comprise a complex set of technologies, varying perceptions of virtual infrastructures and virtual servers, tend to exist. Ask any VI admin, a sales person, or a customer and you will likely get three different answers. As organizations grow, the degree of specialization typically increases, which augments the number of departments that contribute in the service delivery model. A lack of definitions for input, output and responsibility areas between these interfaces can have a negative impact such as prolonged delivery times and an unclear delivery and pricing model. Another consequence of not defining your services is that someone else will do it for you. This could be the sales department or a solution architect that sell a custom solution due to a lack of existing building blocks. These solutions typically do not scale well and the technical design tends to be less than optimal. Services, whether it be an ‘ESX operations service’ or a ‘virtual Windows server service’, need to be defined, standardized, and published in a service catalogue. Furthermore, there should be a clear distinction between an internal service and an external customer offering. These matters will be addressed in this session as well as different examples of how a virtual infrastructure- and a virtual server service can be defined. This session builds on the theoretical framework of the updated ITIL v3, specifically with a focus on Service Design and the Service Catalogue.

Dilbert strip - Owned! ;-)

2010-04-18T12:33:00.002+02:00

Ahh, that guy's funny ;-)

http://dilbert.com/strips/comic/2010-04-07/

Identifying your WWN id's via ILO

2010-03-31T10:45:00.004+02:00

For the storage department to be able to zone up one or more LUNs to a given ESX host, they need three pieces of information:

ESX host name (FQDN)
WWN id's of the HBA's
If new LUN, then the size of the LUN. If you're zoning existing LUNs, then they need to know the storage group that the host should be added to (this can be done by providing hostname of one or two existing hosts that already have that zoning).

The WWN id can be identified both from the VI client (Configuration -> Storage Adapters) and from the service console. But this can only be done after ESX has been installed.

Sometimes, it can be useful to be able to fetch WWN info before the host has been installed. This way, the storage department can begin zoning right away.

To identify WWN id's from ILO

Log into ILO either directly or via the blade enclosure
Go to the Information tab of your server
WWN id can be found under the info box for your HBA (see screendump below)

Howto: Installing VMware tools in a Linux VM

2010-02-15T14:28:00.005+01:00

Installing VMware tools in a Linux VM take a few more steps than on a Windows VM. This is done the following way (tested on VMware Workstation 7 and Ubuntu Desktop 9.04 VM appliance).

install the guest OS (click here to see if guest OS is supported)
to exit the gui to simulate no X server: sudo service gdm stop and then alt+f1 to get console
right click the VM and choose install/update VMware tools. This will connect the cdrom with the VMware tools ISO file (if files are not already available, they will be downloaded) but you still need to mount the cdrom manually: sudo mount /dev/scd0 /media/cdrom (if folder don't exist, create it first)
copy the tar file to /tmp folder and untar it: tar -xvf VMware-tools-vXX.tar.gz
ls to the untar'ed folder and run vmware-install.pl: sudo ./vmware-install.pl
start the gui: sudo service gdm start or simply startx
verify that VMware tools are running: sudo ps -auxwww 'pipe-symbol' grep vm (look for /usr/bin/vmtoolsd and you will also find the balloon driver vmmemctl). You can also check if the vmtools startup script has been put into the startup folder /etc/rc0.d/

link to VMware KB article on installing VMtools (alternatively this KB article)

Update 2011.10.28: When installing VMware tools in Linux Redhat Enterprise 5.6 the installation failed as it needed gcc and some kernel developer packages. I ran the following commands and then reran the tools installation again after ./vmware-install.pl:

yum install gcc
yum install kernel-devel
yum install kernel-xen-devel

Example of an HA error - and a fix

2010-02-11T09:44:00.004+01:00

The other day, I got an HA error when trying to add a new host into a cluster. It was weird, as the host was identical to the others - same model, same installation procedure, and everything. In VirtualCenter, the error looked like this:

This piece of information did not help much in relation to troubleshooting.

The only thing that was different with the new host was that is was configured from the service console (COS) as its NICs were DOA. I had used my own guide for this, so I thought I was in good shape ;-).

A more descriptive error was to be found in the VirtualCenter agent log file on the host (/var/log/vmware/vpx/vpxa.log). Grepping for the word "error" gave the following output:

errorcat = "hostipaddrsdiffer",

errotext = "cmd addnote failed for primary node: Host misconfigured. IP address of ... not found on local interface"

Earlier on, I had changed the IP address, as the first one assigned was already in use, but I'd forgotten to change the IP address in the /etc/hosts file. After doing that and restarting the network (service network restart), everything worked fine.

As a side node, I can mention that it can be pretty confusing manoeuvering through the various log files. Check this post by Eric Siebert for further explanation of VMware log files on VI3.

Differences between Windows Server 2008, SP2, and R2

2010-02-03T12:42:00.003+01:00

So what are the differences between win2k8, win2k8 SP2, and win2k8 R2? These naming conventions and differences between versions are a constant cause for confusion. So here's the short take:

Win2k8 was first released with SP1. Later on came Win2k8 SP2.

Win2k8 R2 is the new version of the OS that introduces several new features. It has the look and feel of Win7, it is only x64 bit, and Hyper-V Quick migration (~VMotion) is introduced.

There's no SP2 installed on top of win2k8 R2. R2 is a clean install or you can upgrade from SP2 to R2. In any case, the SP2 will disappear and it will only be called R2.

The reason for pointing this out is that it was a bit different with win2k3. Here, you installed SP2 and then you installed R2 on top of SP2 and the result was win2k3 SP2 R2 - so service pack and R2 at the same time.

I found this comparison somewhere and I quite like it (not quite sure how correct it is, though..)

Windows Vista SP1 ~ Windows Server 2008 SP1

Windows Vista SP2 ~ Windows Server 2008 SP2

Windows 7 ~ Windows Server 2008 R2

Finally the VCP4 certification Welcome Kit arrived

2010-01-14T23:31:00.004+01:00

Today, the official VCP4 certification arrived in the mail. Six months(!) did it take for VMware to send the papers. There's no visible sign on the certification to indicate that it was taken as part of the beta exam - except for the date in the lower left corner (July, 16th 2009) indicating that it was achieved before the official release in August 2009.

As a bonus, a free license for Workstation 7 was included in the package which is pretty cool.

P2V pre-migration checklist - and considerations

2009-12-24T12:50:00.007+01:00

My prevoius post was a P2V post migration checklist. This post is a pre-migration checklist which is about all the information that should gathered and checked before doing any P2V conversions.

I have been involved in a number of larger P2V projects (+50 P2V's) and, in my experience, proper planning is a key element for a succesful project. Typically, you, as a VMware- or P2V person, have no real knowledge of the Windows servers to be converted - their just another server. This means that you rely on other people to collect relevant data on your behalf. Such a setup has an important implication. As you have no knowledge of the server, it cannot be released into production by yourself, you should let a Windows guy verify the OS after which it can be handed over for application testing. Resources for both tests should be allocated up front by the project manager and they should be standing by in the agreed maintenance window.

In regards to the length of maintenance windows, we have had the most succes with long time frames during weekends - e.g. 36 hours from Saturday 08.00 a.m. to Sunday 08.00 p.m. Obviously, such a window can be difficult to obtain, but it has two significant advantages: 1) Specifying the actual conversion time can be tricky - it happens that a 30 GB server takes 12 hours to convert for one reason or the other. 2) It is less stressfull to do P2V's during weekends and a long window will let you work at your own pace, Furthermore, conversions can run over night if they have large disks (e.g. + 200 GB).

Now, a few words about the checklist. Over time, it has been gradually extended as we have learned important lessons - some of them the hard way where. For example, a server that hadn't been checked for hardware dongles, then you need to roll back - or e.g. a VLAN that hadn't been properly trunked... A specific list will match a specific scenario so, typically, the list will be modified to some degree for each project. However, a large part of the list will remain the same, so hopefully it can be used for inspiration. We use Sharepoint 2007 to organise the lists. These can be dynamically updated, which is practical when multiple persons have to update at the same time.

Servername
OS type
Server model

Has Capacity Planner run for this server?

# of CPU sockets
# of CPU cores
Amount of physical memory installed
Physical disk capacity (C-drive, D-drive, etc.)

Current CPU usage (preferably from cap. planner)
Current memory usage (preferably from cap. planner)
Current physical disk usage (C-drive, D-drive, etc.)

# vCPU’s that should be assigned
Amount of memory to be assigned to VM
Sizes of vDisks after resizing (C-drive, D-drive, etc. – remember separate .vmdk’s for each logical volume)
Total size of vDisks (then you can sum up total disk capacity needed and ask for storage up front)

Local administrator credentials (local windows accounts are recommended)
“Ipconfig /all” screendump attached to list (this is to ensure you have the right IP and mac address)?
ILO-information (address, credentials) (if you have to do cold migration)
Has server been defragmented (this can significantly speed up conversion rates)?
Has server been checked for hardware dongles?
Has VLAN been trunked?
Do server application licenses have any binding to MAC or IP address?

Remote access type (RDP, Netop)? (for stopping services up front)
Physical server location

Applications on server
What services to stop on server before conversion

OS tester contact info
Application tester contact info (for pre- and post migration test)

Server to be converted by (employee)
Date for conversion
Conversion progress/status (not begun, P2V begun, handed over to OS testing, released to production, etc.)
Has physical server been shut down?
Notes

P2V post migration checklist

2009-12-13T11:29:00.006+01:00

When doing P2V projects, I usually have a short, written checklist on my desk to make sure I remember everything. The list is as follows:

For hot migration:

Disable relevant services on the source machine
When configuring the P2V, don't set the VM for autoboot upon completion
Adjust hardware on the VM before first boot (remove serial ports etc)
Check that VMware Tools installs correctly
Adjust the HAL if needed
Uninstall HP software
Remove hidden NIC's
Set IP - if static IP (Start -> Run -> ncpa.cpl)
Check services.msc to ensure that all automatic services are running (and that you re-enabled the ones that you disabled to begin with)
Shutdown the physical server (shutdown /s /t 0 from CMD)
Ping -t the physical server and when it stops responding, then enable the NIC on the VM
Reboot the VM

After this, I typically handover the VM to the Windows Operations team which check the eventlog and such, and then they hand it over to the application testers before releasing it into production.

VLAN testing in ESX 3.5

2009-11-22T21:46:00.015+01:00

In larger organisations, typically, the network department and the VMware group are seperated in different teams. So as a VMware administrator you need to ask the network department to trunk VLANs to the physical switch ports that your ESX is connected to. It happens that the network department misses a port or a VLAN which means that you can end up with a VM loosing network connection after e.g. a VMotion. Unfortunately, the responsibility can land on the VMware administrator for putting a host into production without testing VLAN connectivity. Unfair, but that's life.

But testing VLANs the manual way is rather time consuming. Especially if you have multiple hosts with multiple nics and multiple VLANs. The number of test cases quickly amount to the impossible. If, for example, you have five hosts, five VLANs and 4 NICs in each host, that means (5 x 5 x 4) 100 test cases.

The traditional way of testing is to create a vSwitch with only one vmnic connected. Then connect a VM on that vSwitch with one of the VLANs. Configure an IP address in the address space of the VLAN and ping the gateway. Do this for all the VLANs, and then connect the next vmnic to the vSwitch and start over.

The following method speeds up VLAN testing significantly (in this case from 100 to 16 test cases). It is not totally automated, but I have found it very useful nonetheless.

The basics of it is that you configure a port group to listen on all available VLANs and then you enable VLAN tagging inside the VM and do your testing from there:

1. Create a port group on the vSwitch with ID 4095. This will allow the VM to connect to all available VLANs available to the host.

2. Enable VLAN tagging from inside the VM. This only works with the E1000 intel driver which only ships with 64 bit Windows. So if you have a 32 bit Windows server, then you need to first modify the .vmx file and then download and install the intel E1000 driver from within Windows (Update: Even for Win 64 bit, you need to download and install E1000 manually. The advanced VLAN option is not included in the default driver). This link describes how this is done. Note that when modifying the .vmx, add the following line:

Ethernet0.virtualDev = "e1000"

Note that if you use the default Flexible nic to begin with, there's no existing entry for the nic in the .vmx, so just add the new entry.

Under Edit Settings for the VM, attach the NIC to the VLAN with id 4095.

3. Now you can add VLANs in the VM. Go to the Device Manager and then Properties for the E1000 NIC. There's a tab that says VLANs (see screendump below). As you add VLANs, a seperate NIC or "Local Area Connection" is created for each VLAN. It is set for DHCP, so if there's a DHCP server on that network it will receive an IP automatically. If not, you will need to configure an IP for that interface manually (e.g. by requesting a temporary IP from the network department.). For quickly configuring the IP, you can run the following command from CMD or a batch (.cmd) script:

netsh int ip set address "local area connection 1" static 192.168.1.100 255.255.255.0 192.168.1.254 1

4. Now we will use the Tracert (traceroute) command to test connectivity. The reason that we can't use Ping is the following: If you have multiple VLANs configured and you ping a gateway on a given VLAN - and the VLANs happen to be routable - then you will recieve a response from one of the other VLANs even though the one your are testing is not necessarily working.

But when using Tracert, then you can be sure that if the gateway is reached in the first jump, then the VLAN works. If the VLAN doesn't work, then you will see Tracert doing multiple jumps (via one of the other VLANs) before reaching the gateway (or it will fail if there's no connectivity at all). You can create a simple .cmd file with a list of gateways that you execute from the CMD prompt. Example file:

tracert 192.168.1.254

tracert 10.10.1.254

tracert 10.10.2.254

See below for example screendump.

Before running the batch script you need to have only one physical nic connected to the vSwitch. You can do this in one of two ways. 1) create a seperate vSwitch and connect only one vmnic at a time. Then you control it from VC. Or 2) you unlink all vmnics but one from the service console (COS) with the following commands:

ssh to the ESX host

esxcfg-vswitch -l (to see current configuration)

esxcfg-vswitch -U vmnic1 vSwitch0 (this unlinks vmnic1 from vSwitch0)

esxcfg-vswitch -L vmnic0 vSwitch0 (this links vmnic0 to vSwitch0)

These commands work instantaneously so you don't have to restart the network or anything. Then you run through the test on one vmnic at a time. When done with a host, you VMotion the VM to the next host in the cluster and continue the test from there.

Howto: Using Find command in Service Console

2009-11-14T01:31:00.004+01:00

From time to time you need to locate stuff in the service console and the only command you got is find. 'Locate' unfortunately hasn't been included in the COS. Typically, I forget the syntax and think of another way of locating files - but actually it's pretty simple.

to use the Find command, do the following:

#find / -name searchstring

(#find 'path' -name 'searchstring')

so, if your looking for sshd_config file somewhere in /etc/ it would look like this:

#find /etc/ -name ssh_config

this will be a search on the complete file name. You can use wild cards as well, e.g.:

#find /etc/ -name ssh_co*

Prerequisites - Capacity Planner analysis

2009-11-10T08:15:00.005+01:00

Before starting a Capacity Planner excercise, there is a number of things that should be in place. The following is typically what I send to customers and ask them to have in place beforehand:

No.	Prerequisite	Status
1.	Account with administrative rights on all servers to be surveyed, so: A Windows account(s) that have administrative privileges on local servers.	Username: Password: Domain:
2.	1 x Windows 2k3 or 2k8 server that we can install the Capacity Planner application on: Windows 2003 SP2 standard with min. 1 GB mem, 1 cpu 5 GB free on D-drive. Can be virtual. Should be joined to the domain where we collect data. We should have RDP-access to this server. This server can be virtual. The server needs internet access as performance data will be uploaded to optimize.vmware.com on port 80 and 443 TCP outbound.	Servername: Specifications: Domain: RDP available: Internet access:
3.	Local windows firewall should be disabled on clients to be surveyed or the following ports must be opened in local firewalls, inbound: TCP/UDP Ports: 135-139 and 445 (They are used for communication between Capacity Planner Data collector and windows hosts).	Firewalls disabled: Or ports opened:
4.	WMI and Remote Registry services should be running on all servers to be surveyed (typically they are running by default)	WMI is running: Remote Registry is running:
5.	A list of servers to be surveyed	List in .csv or .xls:

It is recommended that performance data is collected for a period of minimum 30 days and no less than 14 days.

Gartner strategic tech list 2010

2009-11-05T13:56:00.003+01:00

In 2009, virtualization was way up on Gartner's list of strategic technologies. For 2010, virtualization again sets a big fingerprint on the list. Virtualization has been split up into several sub concepts which are represented indiviually on the list.

Configuration notes for HA

2009-11-05T12:05:00.012+01:00

A while back, we experienced a number of inconvient HA failover false positives where several hundred VMs were powered down even though there was nothing wrong with the hosts. The cause of these incidents were apparently a hick-up in the network lasting more than 15 seconds. To avoid such issues, we decided to disable HA until we were absolutely that we had a proper HA configuration.

In the following, there is a quick guide to the HA settings, that we use. These correspond to current best practice.

For reference, we have used the HA deepdive article from Yellow-bricks and article by Scott Lowe on HA configuration notes.

Das.failuredetectiontime

the default timeout for HA is 15 seconds. Best practice is to increase this to 60 seconds or 60.000 miliseconds. To do this, add the following entry under VMware HA -> Advanced options:

Option: das.failuredetectiontime

Value: 60.000

The input is validated, so if you spell it wrong you will be prompted with an error.

Das.isolationaddress

The default isolation address is the default gateway which is pinged if there is no contact between the hosts. However, the default gateway can be some arbitrary place in the network, so it can sometimes be useful to insert one or more extre isolation addresses. It makes sense to add an IP as close to the host as possible e.g. a virtual IP on a switch.

Option: das.isolationaddressX (X=1,2,3,...9)

Value: IP address

Host isolation response

For fibre channel storage, we choose "leave powered on". In a HA failover situation, the active primary node in the cluster will try to boot the VM on the failed host. However, if the host is not down, there will be a vmfs file lock on the VMs and therefore they can't be restarted. HA will try to restart VMs five times. Worst case scenario is that VMs on a host loose network connection... (in vSphere, default response has been changed to "shut down").

For iSCSI storage and other storage over IP, the best practice isolation response is power off to avoid split brain situations (two hosts having write access to a vmdk at the same time).

Cisco switches and port fast

In a Cisco network environment, make sure that 'spanning-tree port fast trunk' is configured on all physical switch ports connected to the ESX host. This ensures that ports are never in 'listen' or 'learn' state - only in 'forwarding' state. So if e.g. one of the uplinks to the COS goes down, you don't risk an isolation response because the delay to put the other port/uplink into forwarding state is longer than the isolation timeout.

Example on a configured interface on a Catalyst IOS based switch:

interface GigabitEthernet0/1
description #VMWare ESX trunk port#
no ip address switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk

HP Blade enclosures - primary and secondary nodes

Due to the fact that there can be no more than five primary nodes in a cluster, a basic design rule is that there should be no more than a maximum of four hosts in a Blade enclosure per cluster. If five or more hosts (and they all happen to be primary nodes) are located in an enclosure and it fails (which happens...), then no VMs will be started. This matter is explained well in the Yellow-bricks article mentioned above. Furthermore, clusters should be spread over a minimum of two enclosures.

Howto: Permission wars in VI3

2009-10-14T15:05:00.011+02:00

UPDATE: This setup doesn't entirely work. Templates aren't visible to the users...

This past week, I have been working on an interesting problem. A new internal customer wanted a development environment where they could free hands to deploy and delete VMs, take snapshots etc. To more or less have free hands and the VMware team should provide the virtual infrastructure as a service.

Now, from a virtual infrastructure operations perspective, to give a customer that much freedom is a bit of an administrative nightmare. For example, how do you ensure that a cluster is not overcommitted and how to make sure that all servers are properly registered in the CMDB.

To address the most important issue - from a technical perspective: The customer should not be able to overcommit the cluster. If they have that possibility, then we can't do maintenance, there won't be full failover. The obvious way to go about it is to enable HA and then to check the 'Prevent VMs from being powered on if they violate availability constraints'. However, HA does not have the most sensible way of calculating HA slot sizes and if you only have two hosts in a cluster, then you risk not being able to deploy a new VM even though there are plenty of resources in the cluster.

A colleague of mine suggested that I create a root resource pool in the cluster and then add permissions only on that resource pool and not on the host, cluster, or datacenter level. In theory, this is a pretty good idea, as you can set a hard limit on the resource pool for memory usage (which in my experience is the typical, visible, limiting factor in the cluster). In this case, I set a limit of 50% of available memory and then made the resource pool non-expandable. The resource pool limits in relation to actually used memory - not what is assigned to the VMs, see below.

I created a role similar (I think ;-)) to virtual machine administrator, which can more or less anything at the virtual machine layer (deploy, delete, change, snapshot, mount ISO's etc.) and added this at the resource pool layer. When I started testing, I discovered a number of issues. First, I couldn't create a VM, I couldn't delete a VM, and I couldn't browse the datastore from the VM summary page. But these permissions were already given to the role. If the same role was applied to the cluster or datacenter level, then it worked fine. So it makes a difference at which level the permisssions are applied.

If I apply the role at the cluster level, then everything works in an acces rights perspective, but then the role have too many permissions. Then, they can deploy servers directly in the cluster and will not be forced to deploy into the root resource pool. And then control is lost.

The only way I could work around this issue was to create two seperate role with two different permission sets and then apply them at two different levels of the datacenter.

The first role has very few permissions and is applied at the datacenter level (do not propagate) (this could also be at cluster level, but currently I only have one cluster in the datacenter...). The second role is the actual role that I created in the first place. This role was applied to the cluster level (propagate rights) where a hard limit has been defined for memory.

Below is listed the permission mapping that I have used for both roles.

With setup, the user is completely locked down, so they can only deploy servers in the defined resource pool and they will not be allowed to overcommit. If they do, the VM's won't be able to power on.

In relation to snapshots and running out of space on the LUN, this problems still persists but will not be addressed in this article.

Role 1 (do not propagate rights) - to be applied at datacenter level

Virtual Machine.Inventory.Create

Virtual Machine.Inventory.Remove (otherwise one can’t delete VM from disk)

Virtual Machine.Configuration.Add New Disk

Datastore.Browse Datastore (to be able to browse datastore from VM summary view)

Role 2 (propagate rights) - to be applied at resource pool level

Datastore.Browse Datastore

Datastore.File Management

Virtual Machine.Inventory.Create

Virtual Machine.Inventory.Remove

Virtual Machine.Inventory.Move

Virtual Machine.Interaction.Power On

Virtual Machine.Interaction.Power Off

Virtual Machine.Interaction.Reset

Virtual Machine.Interaction.Answer Question

Virtual Machine.Interaction.Console Interaction

Virtual Machine.Interaction.Device Connection

Virtual Machine.Interaction.Configure CD Media

Virtual Machine.Interaction.Tools Install

Virtual Machine.Configuration.Rename

Virtual Machine.Configuration.Add Existing Disk

Virtual Machine.Configuration.Add New Disk

Virtual Machine.Configuration.Remove Disk

Virtual Machine.Configuration.Change CPU Count

Virtual Machine.Configuration.Memory

Virtual Machine.Configuration.Add or Remove Device

Virtual Machine.Configuration.Modify Device Settings

Virtual Machine.Configuration.Settings

Virtual Machine.Configuration.Change Resource

Virtual Machine.Configuration.Reset Guest Information

Virtual Machine.Configuration.DiskExtend

Virtual Machine.State.Create Snapshot

Virtual Machine.State.Revert to Snapshot

Virtual Machine.State.Remove Snapshot

Virtual Machine.State.Rename Snapshot

Virtual Machine.Provisioning.Customize

Virtual Machine.Provisioning.Clone

Virtual Machine.Provisioning.Create Template From Virtual Machine

Virtual Machine.Provisioning.Deploy Template

Virtual Machine.Provisioning.Clone Template

Virtual Machine.Provisioning.Mark as Template

Virtual Machine.Provisioning.Mark as Virtual Machine

Virtual Machine.Provisioning.Read Customization Specifications

Virtual Machine.Provisioning.Allow Virtual Machine Download

Virtual Machine.Provisioning.Allow Virtual Machine Files Upload

Resource.Assign Virtual Machine to Resource Pool

Resource.Migrate

Resource.Relocate

Howto: Check if SAN cables are connected in ESX

2009-10-08T16:54:00.006+02:00

When installing an ESX host and you have someone other than yourself taking care of the cabling of the host, it is very handy to be able to check wheather this has been done properly. You want to be able to verify that the HBA's have been physically connected to the fabric switches with fibre cables.

Ssh to the ESX host

ls to the /proc/scsi/qla2300 folder (if it's a Qlogic HBA...)

In this folder there are a number of text files named with the numbers 1-x corresponding to the number of HBA ports in your ESX.

Cat the files one at a time:

#cat 1

#cat /proc/scsi/qla2300/1

look for the following line in the files:

Host adapter:loop state=READY, flags= 0x8430403

If it says READY, the HBA has been physically connected to the fibre switch. If it says DEAD, then it is not.

VTSP 4 certified

2009-10-02T00:02:00.005+02:00

Today, I passed the VTSP 4 (VMware Technical Sales Professional) certification. Apparently, for your company to keep VMware Enterprise Partner status, a minimum of 2 x VCP's, 2 x VSP's, and 2 x VTSP's are required. We have the first two accreditations well covered but needed the VTSP's - so I had to take one for the team together with a couple of the other guys ;-)

The achieve this certification, you need to pass six online tests which you can take at your own pace. These are available through Partner Central. There are self study guides with each test. We received a nice and sweet offer from our distributor to get a two day training session so we got it handled quick and easy...

Supported image formats for Converter 4 Standalone

2009-09-21T15:23:00.003+02:00

VMware Converter 4 Standalone supports the following image formats (see screen dump below). Hyper-V is supported but conversion will have to be done as a physical server (see link)

http://www.vmware.com/products/converter/faqs.html

Problems showing performance stats in VC after DB upgrade

2009-09-11T15:09:00.002+02:00

We had an incident the other day about a VirtualCenter v2.5 U4 not showing performance statistics for the VM's. It was possible to see live stats for e.g. CPU and memory usage. But changing the chart options to 'weekly' or 'monthly' resulted in a 'Performance data is currently not available for this entity'.

Recently the backend SQL Express server for this VC had been upgraded to a SQL 2005 Standard edition and this was the reason for the error.

In the SQL server, there are three stat-roolup-jobs (which are the ones creating the perf stats in VC) which were not automatically created during the upgrade. These had to be added manually following this KB article from VMware. These are:

Past Day stats rollup
Past Week stats rollup
Past Month stats rollup

After adding the jobs and waiting a couple of hours for all of the jobs to have run, everything worked just fine (The VC DB is called VCDB - UMDB is for Update Manager).

Below is a screendump from the SQL Management Studio after the jobs had been added:

Howto: Removing a disk from a VM - howto identy the right disk?

2009-09-11T12:05:00.013+02:00

From time to time, we need to remove disks from a VM. If there's only two or three disks attached to the VM, it's typically not a problem figuring out which one to remove e.g. if the disks have different sizes. But if you have seven or eight disks and they are the same size, then it's a bit more tricky - let's say if you're asked to remove the 'E-drive'. Under 'Edit Settings' for the VM, the disks only have a number which does not necessarily correspond with anything within the VM.

So how to identify exactly which disk that corresponds with a given volume within Windows?

The match can be made by looking at the SCSI target ID for the disk - this can be identified both in WIndows and under 'Edit settings' for the VM (A VM can have four SCSI controllers with up to 15 disks on each controller, so a maximum of 60 disks per VM).

To identify SCSI target ID within the VM:

Go to Computer Management -> Disk Management

Right click a disk and choose Properties

On the General tab you will see the Bus number (SCSI controller) and the Target ID (SCSI target ID), note the number - in this case below the ID is 4.

To identify SCSI target ID from the VI client:

Now go to 'Edit Settings' for the VM under and locate the disk with the corresponding target ID (see Virtual Device Node for the disk). Make sure the that the controller number and SCSI ID is the same. In this case it is Hard Disk 5 that have SCSI ID 4.

Shut down the VM to remove the disk.

SVMotion GUI plugin for VI Client in VI3

2009-09-10T10:19:00.005+02:00

Lost Creations has made this very popular GUI plugin for doing SVMotion from the VI Client. It has been out there for quite some time, so this post is merely for my own reference (I actually thought I had posted about this before...)

It's absolutely a 'must have' tool for daily operations of the virtual infrastructure.

http://lostcreations.com/viplugins

(Update 2011.01.05: Use this link for download in stead)

Go here for installation guide.

SVMotion with .vmdk's on different LUN's

2009-09-10T09:52:00.003+02:00

Yesterday, I had to extend a number of disks on a VM. There were about seven .vmdk's spread over three different LUN's which were all out of space. In VI3 there's really no good way to increase a LUN (unless you use extend, but don't), so to increase the disk sizes of the .vmdk's, a larger LUN had to be created onto which the .vmdk's could be moved before extending them.

The storage guys create a 1 TB LUN for the VM. So, I wanted to use SVMotion to move the .vmdk's one by one to the new LUN. If you start out with a disk that is not the primary, or OS, disk you will get an error (I'm using the GUI plugin from Lost Creations), so you can only move the primary disk. However, when you move that primary disk, all of the .vmdk's attached to that VM will be moved with the VM at the same time and will be placed on the target LUN.

So when SVMotioning, all .vmdk's attached to that VM are moved at the same time. Therefore, make sure to have enough space on the target LUN.

I passed the VCP 4 beta exam!

2009-09-05T19:18:00.004+02:00

Hurrah! This Monday I received an email notification from VMware stating that I have passed the VCP4 beta exam on vSphere 4 that I took on July, 16th. For that I got a fancy little VCP button to wear at VMworld ;-) The email notification only mentioned that I passed and then a formal score report will be sent next week...

VMworld 2010 Europe - in Copenhagen!

2009-08-28T15:56:00.004+02:00

I just saw on ntpro.nl today that VMworld Europe 2010 has been moved from Cannes to Copenhagen (11-14th October). That really warms my heart ;-) One should think, however, that it would be a bit more exotic to have it in Amsterdam in stead - also because VMware apparently is moving very fast there - but maybe it's a question of facilities.

Anyways, it'll be a nice and short 20 minutes trip next year ;-)

Access rights and permissions in vSphere

2009-08-11T13:25:00.009+02:00

In more than one instance, I have experienced a situation where we had issues with managing permissions in both vCenter (vSphere 4) and VirtualCenter (VI3). The issue is that a user loses access rights when a group to which the user belongs is added with less permissions.

An example could be that a given user, 'UserA', has administrator rights at the top level (Hosts and Clusters) and then at a lower level (let's say at Datacenter level), a given security group, Group1, in which UserA exists is given, let's say, 'Virtual Machine User' rights. This will decrease the permissions for UserA on that datacenter to only Virtual Machine User in stead of Administrator - he cuts the tree under himself, so to speak.

The consequence can be that in stead of risking this scenario of suddenly losing access rights when groups are added, then security groups are not used at all, only single users are added. This is not a problem when only a few users needs acces to the vCenter or VirtualCenter. However, if many users need access, e.g. 20-40 employees, it gets rather complex to manage.

To be absolutely sure how these permissions work, I have done a bit of testing on both vCenter and VirtualCenter.

Test cases

First of all, permissions seem to work identically in both versions, that is VI3 (VirtualCenter) and vSphere (vCenter). Furthermore, when permissions are changed in vCenter, then they are applied more or less instantaneously. So if you change or configure permissions for a user that has the vSphere client open, then the changes will appear to the user at the same time while he has the vSphere client (or VI Client) open (this makes it nice and easy for testing purposes, by the way...)

If the administrator role is assigned to UserA at the Hosts and Clusters level, and then he is assigned less permissions at a lower level (e.g. at a given Cluster), then the less permisssions on that lower level will win.

It works the same way the other way around, if UserA has 'Read only' on Host and Clusters and Administrator rights at a given Datacenter, then UserA will have full rights on that Datacenter and read only on the rest of the virtual environment.

If UserA has Administrator rights at the Hosts and Clusters level and at the same time a group to which UserA belongs is added with Read only to the same level - the interesting question is which of the two different permision levels will UserA be granted, Administrator (as a single user) or Read only (as he belongs to the group)?
The answer is that the highest defined permissions defined at a given level for a user will win. In the case UserA will have administrator rights at hosts and clusters level.

Administrators group

Another thing to be aware of is that Windows Administrators on the vCenter server are automatically added as administrators in vCenter. If you do not intend to give all of your Windows admins full acces to your VMware environment, then remove the 'Administrators' group from vCenter (in stead, you can add the local administrator user a an administrator in vCenter, so you have the possibility to log in with a local account should AD fail..)

Security groups or Distribution lists

Only security groups defined in Active Directory (AD) can be used as groups in vCenter. Distribution lists won't work.

Recommendations for managing users

In regards to the use of groups for managing users in vCenter, I recommend that groups are used at the hosts and clusters level (of course, this can vary greatly depending on your setup). For example, you could have three groups:

VMware admins (Administrator)
VM admins (Deploy/destroy rights, change VM specs, etc.)
Windows admins (console access to the VMs, similar to ILO access on physical servers)

Even though a VMware admin belongs to several groups, as long as these are defined at the same level, then he/she will retain administrator rights.

By using security groups, then the VMware admins won't have to manage user administration on the VMware environment. When a user is added to a given group in AD (this should be handled by your user administration department or system), then he automatically gets access to vCenter.

For more info, see Basic System Administration guide pp. 213-230 on VMware.com.

Expert level on Communities - Whoop!

2009-07-27T19:03:00.003+02:00

That's right! Today, I reached Expert level (750 points) on the VMware Communities forum. I have been active on the english forums for about four months now, and that is where I have collected most of the points. Next level is Master and it requires 2000 points so that's propably going to take a while to reach ;-)

Click here for link to overview of the community levels.

Resizing disks in VMware Workstation

2009-07-27T12:58:00.004+02:00

If you want to increase the size of a virtual machine (VM) in VMware Workstation, you can use the command line tool, vmware-vdiskmanager, from a command prompt. The command can be executed from the VMware Workstation folder under Program Files\VMware. The VM should be powered off.

The following command will increase the size of the virtual disk to 30 GB. In this case, the .vmdk file resides on a network share.

C:\Program Files\VMware\VMware Workstation>vmware-vdiskmanager -x 30GB "\\FILESERVER\folder-X\My Virtual Machines\testserver\testserver.vmdk"

This will work both on a disk where all space has been allocated and disk that are allowed to grow.

VLAN trunking / grouping in distributed virtual switch

2009-07-27T08:43:00.006+02:00

In vSphere, there's a new networking feature which can be configured on the distributed virtual switch (or DVS). In VI3 it is only possible to add one VLAN to a specific port group in the vSwitch. in the DVS, you can add a range of VLANs to a single port group. The feature is called VLAN trunking and it can be configured when you add a new port group. There you have the option to define a VLAN type, which can be one of the following: None, VLAN, VLAN trunking, and Private VLAN. But this can only be done on the DVS, not on a regular vSwitch. See screendumps below (both from vSphere environment)

Links to VMware documentation

2009-07-23T18:27:00.008+02:00

I always seem to forget where VMware has placed their documentation and in which docs to look for what documentation (maybe it's just me). So here's a few links:

vSphere main documentation:

http://www.vmware.com/support/pubs/

vSphere Compatibility Matrixes

http://partnerweb.vmware.com/comp_guide2/sim/interop_matrix.php?

vSphere Upgrade guide:

http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html

VI3 documentation:

http://www.vmware.com/support/pubs/vi_pubs.html

Hardware Compability List (HCL):

http://www.vmware.com/resources/compatibility/search.php

VI3 and vSphere patch download:

http://support.vmware.com/selfsupport/download/

Supported guest OS'es

VMware compability guide -> Guest OS

VCP4 Beta exam

2009-07-16T15:30:00.003+02:00

Today, I took the VCP4 Beta exam. And last chance to take is tomorrow. I was lucky to get an invitation, as I fulfilled the two prerequisites: 1. to be a VCP and 2. to have participated in the vSphere 4 beta program. However, I had my doubts about taking the exam as it was announced rather suddenly and it gave me less than two weeks to study for it. But I thought, what the heck, let's study hard for two weeks and then see what happens...

The exam was quite tough, there were more questions than there will be in the final test. Because I'm a non-native English speaker, I got an extra 30 minutes. But still, 270 minutes is not much for 270 questions! That's one question per minute for 4½ hours straight and no breaks... The exam is covered by the NDA so I can't go into details, but I'm glad I reviewed the configuration maximums and then I should have spent more time on resource management (resource pools), iSCSI, paravirtualization, NPIV, and storage.

For study material, I've used:

What's New in VMware vSphere 4 book (that you get on the what's new training course)
VCP4 exam blueprint
vSphere documentation bundle
misc info on professionalvmware.com
Simon Long's practice exam

And now, we just have to wait 6-8 weeks to get the results...

Going to VMworld 2009!

2009-06-21T01:35:00.003+02:00

Oh happy day! Yesterday, we got final confirmation that a colleague and I are going to VMworld in San Francisco in August. Due to the economic crisis and all it has been a bit of a struggle to get the approval, so it was quite the relief.

I was in Las Vegas last year for VMworld 2008 and it was cool so I'm really looking forward to San Francisco this year.

Howto: Getting the Navisphere Agent for ESX Server

2009-06-20T19:39:00.008+02:00

There are several post in the forum about where to download the EMC Navisphere agent. Navisphere is an agent that you install in the Service Console on your ESX Server which helps to manage EMC Clariion storage systems. Click here for more info.

The agent is not publicly available for download. If you have a partner login, then I believe you can download it at http://powerlink.emc.com/ .

The way to go to get the agent is via your storage department. Either they can get the login for you or have them contact EMC, then they will send the software. Navissphere is shipped together with the Clariion storage systems on the Navisphere Server Support CD (see this article page 16). But contact EMC if you want to be sure to have the latest version.

In this document on page 7, it is stated that Navisphere v6.22 is compatible with ESX v3.5

ESX 4.0 in Workstation - requires Intel-VT

2009-06-17T00:09:00.010+02:00

I have been running ESX 3.5 and ESX 4.0 in VMware Workstation 6.5.1 for a while on my Lenovo T61 from work without any problems. A prerequisite for doing this, at least for ESX 4.0 (an probably also for Hyper-V) as it runs 64-bit, is that the CPU supports virtualisation mode - which in the Intel terminology is called Intel-VT - an which has to be enabled in the BIOS. The T61 is about one year old and has Intel-VT, so I thought that it was standard on all newer Intel processors. But oh-no, this is not the case. I recently purchased a Dell Studio 17 for private use with a Intel Core Duo 2 T6400 processor and I thought that I was in the good house. But - no Intel-VT support. Everything else was in order, 4 GB of memory, Windows 7 64-bit and so on. This was a bit disappointing. If your're looking to buy a new laptop, then check that this feature comes with the CPU. I found an article on ZDnet which lists a number of processors and wheather they have Intel-VT enabled.

The following has been copied from the ZDnet article. YES means that the CPU type supports Intel-VT:

Update: Here's a link to Intel's page with Intel-VT technology list

vCenter Converter Standalone 4 - ports used

2009-06-13T16:23:00.004+02:00

We're doing quite a few P2V conversions at the moment, and that means that we see all kinds of weird errors, conversion failures, and connection issues. P2V is definitely not an exact science.

One thing that is recommended to have in order is that proper network ports are opened.

VMware has written a good KB article that explains which ports are used.

If you have server with Converter Standalone installed on it, and you have trouble connecting to the source physical computer, then first make sure that Windows Firewall is disabled. If that doesn't work, then install the Converter application directly on the source computer. Then you will need outbound 443 TCP connection to vCenter (former Virtual Center) (it's assumed that port 443 TCP is open inbound on the vCenter server, of course).

To test if ports are open, open a CMD prompt and run following command:

telnet 'vCenter ip' 443

(without the ' ') If the DOS prompt goes black, then the connection is good. Othervise you will get a 'can't connect' or something similar)

If you P2V directly to an ESX server, then ports 902, 903, and 443 TCP are used.

If you, for some reason, can't get port 443 opened, then a workaround is as follows:

Install the Converter directly on the source system
If you have an existing test VM in the same IP range, then create a new disk and attach that to the test VM.
Make a Windows share on the new disk
From the Converter choose to export to standalone virtual machine in Workstation format and then coose to place files on the share just created
After export, change the VLAN to an IP range that doesn't have any firewalls blocking
Import the VM from within vCenter

P2V of domain controller

2009-06-11T22:31:00.007+02:00

Summary: Cold clone P2V of domain controllers works just fine.

We had to migrate two root domain controllers the other day at work. I knew that domain controllers in particular can give you trouble when being converted / migrated, so I researched it a bit and found a useful article on yellow-bricks.com which linked to a very good VMware KB article . This KB recommends that in stead of migrating, then deploy a fresh VM and do a 'dcpromo' and then shut down the physical server after. I like this way as it moves the responsibility away from the VMware team and over to the application responsible.

However, we did not have enough time to do the recommended solution, so we whent for P2V. We did cold clone because hot migration is likely to go wrong and it is not supported by Microsoft.

There were FSMO roles on the DC's, so before we began, we had the AD guy move all the roles over to one of the servers. Then we took the other one down and P2V'ed it. We resized the disks to save SAN space which was not a problem. When it came back up, the AD guy tested and then moved FSMO roles over to the migrated DC. And then we migrated the other one. After both had been migrated, the AD guy tested again.

If your responisbility area does not cover the application layer, which it does not for me in this case, then arrange for an application responisble to test the app before it is released into production. It may sound banal, but it is sometimes overlooked when the pace is fast and only basic OS testing is done.

Time synchronization

There are several ways of setting up time synchronization. One important point is that there should be only one source for synchronization for all the DC's. There's a feature in VMware tools, where you can synchronize the VM against the ESX - this we did not use. We let Windows take care of the synchronisation. If you have a mixed environment of DCs (bare metal and virtual), then you can let a bare metal DC sync to an external source, and then let all the other DC's sync to the bare metal DC.

We had the PDC emulator sync with a dedicated physical NTP server, and then let the second DC sync with the PDC emulator. The ESX servers sync with the physical NTP server - but no synchronization between VM and ESX server. Read this article for further info on time sync.

Update: In a KB article (KB 888794) from Microsoft about considerations when hosting DC's in a virtual environment, there is one important paragraph about forced unit access (FUA) which has resulted in some confusion. The paragraph states:

"If the virtual hosting environment software correctly supports a SCSI emulation mode that supports forced unit access (FUA), unbuffered writes that Active Directory performs in this environment are passed to the host operating system. If forced unit access is not supported, you must disable the write cache on all volumes of the guest operating system that host the Active Directory database, the logs, and the checkpoint file. "

According to VMware, forced unit access (FUA) is supported on VMware. Here's the answer from VMware technical support:

-----Original Message-----
From: VMware Technical Support [mailto:webform@vmware.com]
Sent: 24. februar 2010 11:25
To: (Jakob Fabritius Nørregaard)
Subject: Re: VMware Support Request SR# 1490632591

** Please do not change the subject line of this email if you wish to

respond. **

Hello Jakob,

Forced Unit Access is supported by VMware. A large number of customer's have virtualized Domain Controllers which is evident in the community forums.

Thanks & Best Regards

Derek Collins

Technical Support Engineer

VMware Global Support Services

1-877-486-9273

VMware Technical Support Knowledge Base

http://kb.vmware.com/kb"

Howto: 101 Scripting ESX server installation on vSphere 4

2009-06-06T00:11:00.016+02:00

I have been wanting to look into scripted ESX installations for a while now but haven't gotten around to it untill now. At first glance it looks a bit complicated - there a several much used deployment tools around (e.g. EDA and UDA), people are posting bunches of deployment scripts etc. I wanted to know the absolute basics - what is the simplest way to script an ESX installation?

First off, I recommend that you download the ESX and vCenter Server Installation Guide and read pp. 43-58 on scripting installations. This documentation helped me to get started more than posts on the web.
On ESX 4, there are two built-in scripts that you can run when you boot the installation CD: 'ESX scripted install to first disk' and 'ESX scripted install to first disk (overwrite VMFS)'. But that's a little boring as these scripts can't be modified.

In stead, we can let ESX server generate a script for us based on your own installation. I like this way as it simplifies things compared to the very comprehensives scripts out there - and it fits to your environment. Better to have a simple script that works than to have a do-it-all script that doesn't. You can always expand it later.

When you install ESX 4 in the default graphical mode, then a Kicstart script (ks.cfg) with your specific settings is generated and placed in the /root/ folder of your ESX installation. Make a copy of this file, as this is the one we will be using as our base script.

This script is to be copied to the root of the installation ISO. To do that, you need an ISO modifying tool like MagicISO (you need to pay 29$ to make ISO's larger than 300 MB). Open the ESX 4 installation ISO in MagicISO and copy the ks.cfg file into the root of the ISO.

Now, boot your server with the ESX ISO. When the first installation screen shows as below, then hit F2 to get 'other options'. Then shift down to the 'ESX scripted install using USB ks.cfg'. We will not be installing from USB, we will just use the command as a template and modify it to get the ks.cfg script from the CD in stead.

Modify the boot options command like this:

Boot options initrd=initrd.img vmkopts=debugLogToSerial:1 mem=512M ks=cdrom:/ks.cfg quiet

That's it. This will do a basic scripted installation of the ESX 4 server...

Mini troubleshooting: I tried to reinstall the ESX server that I had already installed, which means that there is already a VMFS partitioned disk on the server. So the clearpart command needed the --overwritevmfs flag to work. Furthermore, in the partitioning section I had to comment out some lines and in stead uncomment the 'part' commands with the --firstdisk flags.

I have pasted the basic script below for reference.

----------------sample ks.cfg----------------------

# Don't edit script in notepad or Word. Use Notepad++ or like app

accepteula

keyboard dk

auth --enablemd5 --enableshadow

#I have added the '--overwritevmfs' flag which is

#necessary when reinstalling an existing ESX

clearpart --overwritevmfs --firstdisk

install cdrom

#The encrypted password is taken from the original

#graphical install

rootpw --iscrypted $1$k364YM8i$CyveR0PWuw294uX8HLzcE0

timezone --utc 'Europe/Stockholm'

network --addvmportgroup=true --device=vmnic0 --bootproto=dhcp

part '/boot' --fstype=ext3 --size=1100 --onfirstdisk

part 'none' --fstype=vmkcore --size=110 --onfirstdisk

part 'Storage1' --fstype=vmfs3 --size=8604 --grow --onfirstdisk

virtualdisk 'esxconsole' --size=7604 --onvmfs='Storage1'

part 'swap' --fstype=swap --size=600 --onvirtualdisk='esxconsole'

part '/var/log' --fstype=ext3 --size=2000 --onvirtualdisk='esxconsole'

part '/' --fstype=ext3 --size=5000 --grow --onvirtualdisk='esxconsole'

%post --interpreter=bash

----------------sample ks.cfg EOF----------------------

New memory hot add feature in vSphere 4

2009-06-02T21:07:00.006+02:00

In the new vSphere, there is a new feature - hot add memory and CPU. This is very cool as downtime can be avoided which is, at least in my company, very helphul during daily operations. To enable these features, do the following:

Make sure the virtual machine hardware is upgraded to version 7 (right click the VM, choose 'Upgrade Virtual hardware', see below).
Go to edit settings for the VM, then the options tab -> Memory/CPU hotplug and enable the two features (see below).

This is all very fine. However, for the features to work, you need a compatible VM OS. And Win2k3 Standard isn't supported. For Windows, it is primarily Enterprise Edition and Datacenter Edition that will work for win2k3. For Win2k8 at least hot add memory will work for all editions 32-bit and 64-bit. See the VMware Compability Guide or here for compatible guest operating systems.

iSCSI on a Windows box with Starwind

2009-06-01T01:21:00.006+02:00

If you want to run shared storage in e.g. a test setup, then Starwind's iSCSI application can be recommended. Earlier, I have tried Openfiler (iSCSI in a linux distro VM) which works fine, It's not too complicated to configure, but still it's much easier with Starwind iSCSI in a Windows environment. The application is free but there's a 2 TB storage limit.

Download Starwind Free here.

Starwind installation guide. It's easy to install, just go next next done. Load the serial key, connect to the localhost (user: test, pw: test), add a new device, and then follow install guide page pp 5.

For a guide on how to configure iSCSI in ESX 3 and 4, click here

A typical test setup could be one physcial host with 8 GB of memory and one quad core cpu. Check vm-help.com for a list of compatible whitebox ESX hardware.

From ESX 4 it's possible to run ESX as a VM within an ESX. Even VMotion will work (go to vcritical.com for a demonstration). And if you have a VM with an iSCSI target running on it, then you have a full enterprise setup running on one box.

Top VMware blogs and sites - in short

2009-05-30T10:08:00.007+02:00

If you're new to VMware it can be a little overwhelming to figure out what and where to get info to get started. The web is full of excellent blogs and sites with VMware stuff, not to mention the VMware communities. I've come across numerous sites that list top blogs and sites and they are very rich on info but maybe it's a little too much if you're just starting out. So in stead of going full throttle, I will list just a few of the most popular blogs and sites. If you follow these, then you'll get a good picture on what's going on. I recommend using an RSS reader like Netvibes for keeping the overview. Here's the sites I recommend:

Blogs

http://www.yellow-bricks.com/ - Duncan Depping, VMware Employee. One of the most popular VMware blogs.

http://www.ntpro.nl/blog/ - Eric Sloof, has his own consulting firm. Dutch guy. The Netherlands is probably one of the most advanced European countries in regards to VMware.

http://vmetc.com/ - Go ugly green with Rich Brambley

http://virtualgeek.typepad.com/virtual_geek/ - EMC guy. Writes some very technical an in depth posts

Communities

VMware Communities - an account in the communities is a must. People here are very fast to respond and very helpful. Try google before you post, though...

Sites

vSphere Land - this site has two very cool subsites. vLuanchpad is a comprehensive list of top blogs and sites. Use this if you want to get further inspiration. And then the Top 10 lists where I especially like the one with Tools.

VMware icons for Powerpoint

2009-05-24T22:54:00.006+02:00

At the VMware VIOPS page you'll find a whole package of icons for making snazzy Powerpoint presentations. Basically they're the same as the Visio stencils mentioned in an earlier post but just in a PP friendly format for easy copying and pasting. Go here for download.

Visio stencils for VMware - icons and diagrams

2009-05-24T22:21:00.011+02:00

When making visio diagrams of virtual infrastructures, there's plenty of icons for depicting the IT infrastructure. But when it comes to VMware and virtualization specific icons, then Visio is all out. On the Vmware VIOPS page, you can dowload the official VMware stencils (Udate 2010.01.13: The link seems to be down... Here's an alternative link) from VMware which will give you all you need for making boss friendly diagrams. Look for the .zip attachments at the bottom of the page. See sample below. (Update 2010.09.06: Here's a link to updated stencils from VMguru)

A first look at installing ESX 4.0 GA

2009-05-21T12:55:00.010+02:00

As most people probably know by now, vSphere 4.0 was released in GA earlier today (2009.05.21). To try it out, I have installed ESX 4.0 in Workstation 6.5.1 on a Lenovo T61 - and it works fine. For those who haven't had a chance to play with it yet, here's some screenshots of the installation process, see below.

Workstation VM settings:

OS type: Redhat Enterprise Linux 4 64-bit
Mem: 2 GB (that was minimum on the beta)
Network: Bridged (tick the "Replicate physical network connection state)
Processor: 1, choose Intel-VT or AMD-V as preferred mode

P2V - Error with NIC after migration with static IP

2009-05-20T14:04:00.005+02:00

When doing a P2V and the server has to have a static IP address after the migration, then you may recieve an error message stating that there's an IP conflict and the ip is already configured on an existing NIC - even though only one VMware NIC is visible in network connections.

The reason is that the physical NICs have not been entirely uninstalled in Windows, they still exist in the device manager as hidden devices. Do the following to uninstall the hidden NICs:

Open a command prompt and type the following commands:

set devmgr_show_nonpresent_devices=1
start DEVMGMT.MSC

Click ‘View’ and then click ‘Show Hidden Devices’.
Expand the Network Adapters tree and right click the dimmed network adapter and click ‘Uninstall’ (You may also see a hidden 'RAS async adapter' device under NICs. This cannot be uninstalled. However, it doesn't matter as it doesn't influence the NIC issue, so just leave it).

Now you can configure the static IP with no errors.

Making ISO's with MagicISO

2009-05-17T22:50:00.003+02:00

MagicISO is a good tool for making ISO files. It's easy to use, just choose your files in an explorer like view make the ISO. In the free edition, there is a limitation on the size of the ISO's - around 300 MB. Get it at http://www.magiciso.com/ .

On another note, ISO's is an easy way of transferring files to a VM. It can cumbersome to transfer files to a VM via the network. Sometimes you have to go through one or more jumpstations. In stead, you can make an ISO of the files you want to transfer, and then mount the ISO directly to the VM from within the VI client.

PsTools: Remote execution of commands in Windows

2009-05-17T22:08:00.005+02:00

PsTools is a number of command-line tools that lets you execute useful commands remotely. Below is a list of tools included in the package which is free and can be dowloaded from the MS Technet site .
I have found a number of these tools very helpful:
PsExec: Lets you execute commands remotely. Use this command to start a command prompt:

psexec \\'computer name or ip' -u 'user' -p 'password' cmd.exe

PsKill: If you have a server that are maxing out its CPU and it takes forever to RDP to it, then you can use first PsList to show running processes and then PsKill to stop the process which is using the CPU.

PsShutdown: If you want to reboot a server via RDP it often hangs in the process. And if you haven't got ILO, then your screwed if you haven't got physical access to the server. Normally it helps to RDP to the console (Start -> Run -> mstsc /console) but if not PsShutdown lets you shut down the server or reboot it.

If, for some reason, you can't execute the commands remotely, then a trick is to make an ISO file containing the PsTools and mount that ISO on the VM. That will make the tools directly available on the VM.

Howto: Network configuration from the Service Console

2009-05-06T14:20:00.017+02:00

Configure service console ip from scratch

If you have screwed up your network configuration and you have lost the connection to your ESX host, here's a couple of commands to configure the network settings from scratch (alternatively, you can try the new console-setup tool that has been introduced as of ESX4.0 U2).

First, delete the service console port group (the vswif) and then delete the virtual switch (typically vSwitch0) so you can start from scratch:

esxcfg-vswif -d vswif0
esxcfg-vswitch -d vSwitch0

Then create a new vSwitch, attach one or more pnics, add a service console port group (vswif), and configure and ip address for the service console:

esxcfg-vswitch -a vSwitch0

esxcfg-vswitch -L vmnic0 vSwitch0
esxcfg-vswitch -L vmnic1 vSWitch0

esxcfg-vswitch -A 'Service Console' vSwitch0

esxcfg-vswitch -v 55 -p 'Service Console' vSwitch0 (adds VLAN55 to the Service Console)

esxcfg-vswif -a -i 10.44.12.2 -n 255.255.248.0 -p 'Service Console' vswif0

Check your results with:

esxcfg-vswif -l
esxcfg-vswitch -l

Edit the host IP address in the /etc/hosts file (use: #nano -w /etc/hosts)

When done, restart the network with:

service network restart

For changing the default gateway, edit the /etc/sysconfig/network file. Or use the command:

esxcfg-route ip-of-gateway

Reboot the server when done.

See which physical nics are connected to a switch
This command can be handy if you're not quite sure which of your physical nics have in fact been cabled to a switch:

esxcfg-nics -l

Retrieve CDP (Cisco Discovery Protocol) info from service console
With this command you can access CDP info from the COS. This is useful if, for example, your network department has not trunked the proper VLANs through and they want to know exactly which switches and ports are connected to your ESX host.

First enable CDP in both listen and advertise mode (default is only listen, so network dep can't see the nics):

esxcfg-vswitch -B both vSwitch0

vmware-vim-cmd /hostsvc/net/query_networkhint more
(insert 'pipe' before the 'more' command, I can't display it here)

look for the following lines:
device = "vmnic0"
devId = "switchname.bla.com"
portId = "GigabitEthernet9/28" (if it says FastEthernet or just Ethernet it is a 100 Mbit and 10 MBit port and the cable should be changed to a Gigabitethernet port)

VLAN / Portgroup replicator tool

2009-04-17T11:50:00.006+02:00

This is not a new tool (ITQ Infrastructure Client), but it is still very useful. Let's say you have to setup six hosts in a cluster and you have to configure two vSwitches on each host with several portgroups or VLANs on each switch. This can take quite some time if you're scripting the post confirguration. And there are plenty of room for error as each host is configured manually.

With this tool, you only need to configure the vSwitches and VLANs on one host manually. Then you can export the network configuration and apply it to the remaining hosts all at once.

With VI4 this feature will be built in with "Host Profiles" but until then, this tool can be used.

Master VCP certification is on the way

2009-04-15T20:57:00.003+02:00

The VCDX certification is rather complicated to acquire. You have to travel to take the exams (enterprise administration and design exam) and you have to submit a written report. This is all a bit too much of a hassle for most people. But now apparently, according to a post on Eric Sloof's site, there's a new VCP Master on the way which is obtained with a VCP and then only the Enterprise Administration exam.

NLB in VMware

2009-04-15T19:49:00.017+02:00

There's a lot of posts in the forums about Microsoft NLB (Network Load Balancing), and it can be quite confusing to figure out what's what. We had problems with it for months before we got it fixed. One of the reasons that it took so long is that there are two ways to go about it and one way is to tweak ESX vSwitch in an ugly way an the other is nice and neat but involves the involvment of the network department - I'll describe the latter solution.

First of let me state: NLB works just fine in VMware. It is not necessary to make any changes on the ESX server or in the vSwitch.

The following steps should be in place:

1. Install and configure NLB on your Windows server like you would in a physical setup.
2. NLB should be configured for multicast
3. A static arp entry needs to be added to the physical layer 3 switch or router that acts as default gateway (note the default gateway from within Windows) for the NLB nodes. If it's a Cisco switch, then the command will be like this (if it's not Cisco, then it may work without adding the arp entry...):

arp <ip of nlb cluster> <mac of nlb cluster> arpa

Adding a static ARP entry to the ARP table means that you're associating a specific IP address with a given multicast mac address. This is not automatically allowed in the Cisco switch/router and therefore such requests are dropped unless manually added.

If the default gateway happen to be a firewall and not a router or layer 3 switch (of the type Cisco PIX or Cisco ASA. Update 2010.10.28: It has been observed that ASA OS version 8.23 requires a static ARP entry. Use command: ”arp <interface> <Multicast IP> <Multicast MAC addr.> alias” Eksempel: arp dmz2 192.168.131.29 03bf.c0a8.831d alias ) then it is not necessary to add the static entry as these firewalls allow adding multicast addresses to the ARP table. If you're not sure weather default gateway device allows adding multicast addresses to the ARP table (meaning that it allows traffic to a specific IP address to be broadcasted to multiple ports on the switch/router), then log into the device and list the ARP table with the following command:

show ip arp

If the multicast mac address exists in the table, then you don't have to make any changes.

The configuration is done directly in Cisco’s ”privileged” mode. See screendumps below (thanks to Kim Rubeck for lending me the dumps):

A good resource on Vmware's site: Link to KB article

Thanks to Kim Rubeck for input!

EVC - CPU compability for VMotion

2009-04-15T19:04:00.012+02:00

EVC (Enhanced VMotion Compability) increases the VMotion possibilities between different processor generations. EVC is introduced from ESX v3.5 u2 and Virtual Center v2.5 u2.

Intel Nehalem processor type is supported from ESX v3.5 u4 (and VC2.5 u2 minimum, I guess)

Link to KB article

Link to Nehalem architecture on Wikipedia

Update 2011.03.18: A BL460cG6 (E5520 Nehalem 45 nm) is compatible together with a BL460cG7 (E5620 Nehalem 32 nm Westmere) in an EVC cluster with Intel Xeon Core i7 (currently second highest level for intel processors) level enabled. However, if Intel Xeon Core i7 32 nm is chosen, then only G7 blades can be used.

Howto: Power off or kill a VM that is stuck

2009-04-13T22:43:00.007+02:00

If you have a VM that is stuck and you can't power it off via Virtual Center, then here's some usable commands for troubleshooting and killing the VM from the Service Console:

Show running VM's:

vm-support -x
vmware-cmd -l

See the current state of a VM:

vmware-cmd /<path-to-vmx-file> getstate

Show running processes related to a given VM:

ps -auxwww grep <name-of-VM>
(Remember the 'pipe' character before grep-command - I can't display here...)

If you see an 'S' next to the process, then it can be killed. If it's a 'D', then the process is defunct and you may have to reboot the ESX server to kill the process.

Kill the proces, and thereby the VM:

kill -9 <process id>

Links to relevant threads:

http://communities.vmware.com//message/1219151#1219151

http://communities.vmware.com/thread/87797

VMware Communities VMTN levels

2009-04-13T20:59:00.004+02:00

Here's the levels for VMware Communities. I always forget where to find them when I need them, so I snatched this from vmware-land.com.

Howto: Extend system partition with Diskpart

2009-04-12T14:44:00.004+02:00

Diskpart is a useful tool for extending logical Windows partitions (typically d-drive) which are not system partitions (c-drive) from within Windows without having to boot on a PXE-cd and rearrange partitions with e.g. Partition Magic. See here for Diskpart instructions.

Here's a workaround for extending the system or boot partition with diskpart (this will only work if the c-drive is the only partition on the .vmdk file). The logic of it is to take the whole .vmdk file and attach it to another VM as a second disk. This way Diskpart will let you extend a system partition:

1. Shut down the VM on which you whish to extend c-drive (c-drive-VM)
2. Create a new temporary VM (temp-VM)
3. Create a new disk on the temp-VM and choose 'Attach existing .vmdk file'. Browse for the .vmdk file of the c-drive-VM
4. Edit settings for the temp-VM and chose the newly added disk and resize it to new size.
5. Boot temp-VM and extend using Diskpart (you may need to assign a drive letter to disk in Disk Management)
6. Shut down temp-VM
7. Delete disk from temp-VM
8. Now the disk has been extended but it c-drive-VM it still shows as the original size. Therefore, delete the disk from c-drive-VM and add it again. This way, the correct disk size will show.
9. Boot c-drive-VM. Done.

Now, maybe this is not a super elegant way to go about it, but if you have a temp-VM ready then it's probably a little quicker than using the old way...

Putting a host in maintenance mode hangs at 2%

2009-04-11T22:57:00.008+02:00

When putting a host into maintenance mode, it happens that the progress hangs, or times out, at 2% and then nothing further happens. VMotion is not commenced for any of the VM's.

There are a number of things, that should be in place before automatic evacuation of VM's will work.

DRS should be set to "Fully automated"
If HA is enabled and only two hosts in the cluster, then the "Allow VM's to be powered on even if they violate availability constraints" option should be checked (see screendump below)
All CD's on the VM's must be disconnected. Eric Sloof from ntpro.nl has made a cool application, vmcdconnected, where a given host can be scanned and all CD's can be disconnected with one click (see screendump below). After scan, all the VM's with a '+' in front of it has a CD-rom connected.

CPU compability tool

2009-04-11T22:21:00.004+02:00

This tool from run-virtual.com can be used to check the CPU compability of your physical servers. On many servers, the VT feature is disabled by default which means that 64-bit VM's cannot run. With this tool, this can be checked without having to boot into BIOS.

Draw a nice Visio digram from your VC using Powershell

2009-04-11T02:32:00.007+02:00

If you have been drawing infrastructure diagrams in Visio of your company setup, you know how cumbersome it can be - especially as the infrastructure changes often. In stead of manually changing your Visio drawings, here's a new tool, that can depict an exact copy of your Virtual Center (now vCenter) or a given cluster - or from a given host - in Visio.

You need to have Powershell and VI Toolkit installed to run script. See here for installation instructions.

1. Go to Virtu-al for further instructions. Download the vDiagram.zip file.
2. Once extracted copy the 'My-VI-Shapes.vss' file to your 'My Documents\My Shapes' folder. If the folder does not exist create it and copy the file in.
3. Run the powershell script (Start-> VMware VI Toolkit -> VMware VI Toolkit ) with the following options:
To diagram the entire Infrastructure:
vDiagram.ps1 -VIServer MYVISERVER or HOST
To diagram a specific cluster use the following:
vDiagram.ps1 -VIServer MYVISERVER -Cluster "Production Cluster"

How to extend D-drive in Windows with Diskpart

2009-04-11T02:08:00.008+02:00

In stead of booting into PXE-boot and starting e.g. Partition Magic for rearranging disks, Diskpart can be used for extending a logical partition which is not the system partition, typically the d-drive.

1. Shut down the VM
2. Right click the VM, choose Edit Settings, click the hard disk, type in new disk size in GB, Click OK.
3. Start VM
4. Start a CMD prompt
5. Run: Diskpart
6. Run the following commands:

List volume
Select volume 2 (if volume 2 corresponds to D drive)
Extend
Exit

7. Done. Reboot to be sure.

How to enable 64-bit in BIOS on HP server

2009-04-11T02:01:00.008+02:00

To be able to run 64-bit VM's in VMware ESX server, then Intel-VT technology needs to be enable in BIOS. Furthermore, to enable EVC (Enhanced VMotion Compability), No-execute memory feature should be enabled, see below.

1. Go to BIOS (press F9 during boot)
2. Go to Advanced Options -> Processor Options -> Intel ® Virtualization Technology
3. Choose Enable
4. Furtermore, to enable VMware EVC, enable 'No-Execute Memory Protection' (just above Intel-VT).
5. Save and exit

NB: All hosts in your cluster should have the same BIOS settings. If not, this can result in VMotion issues.

Understanding the snapshot - how to check size of a snapshot

2009-04-11T01:41:00.011+02:00

When creating a snapshot, the existing vmdk file is locked and a new vmdk is created, a delta file. If there are multiple vmdk's attached to the VM, seperate delta files will be created for each vmdk. If vmdks are placed on other LUNs than where the .vmx file is residing, then all delta files will be placed on the same LUN as the .vmx file. All changes made after the snapshot is taken are added to the new vmdk file(s). The delta vmdk files can grow until they reach the size of the original vmdk file. If a snapshot exists for too long, this can generate problems as the SAN LUN can run out of disk space. If this happens, the VM’s will start to crash. Therefore, as a general rule of thumb, snapshots should not be left unattended for more than one or two weeks unless it is ensured that there is sufficient space on the data store. If the snapshot is needed for a longer period, it is recommended to make a clone instead.

To check the size of the snapshot, simply browse datastore and look for a numbered vmdk file, e.g.

server123-0000001.vmdk

If a second snapshot is taken, it is named:

server123-0000002.vmdk

And so forth…

Below is a number of screenshots where you can see how files are created as snapshots are made:

1. This first sreenshot, VM is just created, no snapshot:

2. Just after first snapshot taken – no further action taken

A new vmdk file is created which is about 18 MB in size when no changes has been made yet. Remark filename, jnrrsnaphosttest-000001.vmdk

3. After installation of a couple of applications

As changes are made, the new vmdk file increases in size. In this case it increases from initial 18 MB to 198 MB. See same file as above.

4. After second snapshot taken:

When a second snapshot is created yet another vmdk file is created (e.g. server123-0000002.vmdk) and so forth...

Boot a server directly into safe mode

2009-04-11T01:24:00.008+02:00

When doing a P2V, it is often necessary to get at VM into safe mode to disable any running HP services and afterwards unistall these HP services (in normal Windows operations). This can either be done by hitting F8 really hard upon boot, or it can be done more smoothly, as hinted by one of my colleagues:

Ones Windows is booted, Start -> Run: msconfig

Tick the /SAFEBOOT option. Reboot the machine, and it will reboot in safe mode. Uncheck this checkmark before returning to regular operations.

Submit a feature request to VMware

2009-04-11T01:21:00.001+02:00

To submit a feature request to VMware, go to the following link:
https://www.vmware.com/contact/contactus.html?department=feature-request