Finally the VCP4 certification Welcome Kit arrived

Today, the official VCP4 certification arrived in the mail. Six months(!) did it take for VMware to send the papers. There's no visible sign on the certification to indicate that it was taken as part of the beta exam - except for the date in the lower left corner (July, 16th 2009) indicating that it was achieved before the official release in August 2009.

As a bonus, a free license for Workstation 7 was included in the package which is pretty cool.

P2V pre-migration checklist - and considerations

My prevoius post was a P2V post migration checklist. This post is a pre-migration checklist which is about all the information that should gathered and checked before doing any P2V conversions.

I have been involved in a number of larger P2V projects (+50 P2V's) and, in my experience, proper planning is a key element for a succesful project. Typically, you, as a VMware- or P2V person, have no real knowledge of the Windows servers to be converted - their just another server. This means that you rely on other people to collect relevant data on your behalf. Such a setup has an important implication. As you have no knowledge of the server, it cannot be released into production by yourself, you should let a Windows guy verify the OS after which it can be handed over for application testing. Resources for both tests should be allocated up front by the project manager and they should be standing by in the agreed maintenance window.

In regards to the length of maintenance windows, we have had the most succes with long time frames during weekends - e.g. 36 hours from Saturday 08.00 a.m. to Sunday 08.00 p.m. Obviously, such a window can be difficult to obtain, but it has two significant advantages: 1) Specifying the actual conversion time can be tricky - it happens that a 30 GB server takes 12 hours to convert for one reason or the other. 2) It is less stressfull to do P2V's during weekends and a long window will let you work at your own pace, Furthermore, conversions can run over night if they have large disks (e.g. + 200 GB).

Now, a few words about the checklist. Over time, it has been gradually extended as we have learned important lessons - some of them the hard way where. For example, a server that hadn't been checked for hardware dongles, then you need to roll back - or e.g. a VLAN that hadn't been properly trunked... A specific list will match a specific scenario so, typically, the list will be modified to some degree for each project. However, a large part of the list will remain the same, so hopefully it can be used for inspiration. We use Sharepoint 2007 to organise the lists. These can be dynamically updated, which is practical when multiple persons have to update at the same time.

• Servername
• OS type
• Server model

• Has Capacity Planner run for this server?

• # of CPU sockets
• # of CPU cores
• Amount of physical memory installed
• Physical disk capacity (C-drive, D-drive, etc.)

• Current CPU usage (preferably from cap. planner)
• Current memory usage (preferably from cap. planner)
• Current physical disk usage (C-drive, D-drive, etc.)

• # vCPU’s that should be assigned
• Amount of memory to be assigned to VM
• Sizes of vDisks after resizing (C-drive, D-drive, etc. – remember separate .vmdk’s for each logical volume)
• Total size of vDisks (then you can sum up total disk capacity needed and ask for storage up front)

• Local administrator credentials (local windows accounts are recommended)
• “Ipconfig /all” screendump attached to list (this is to ensure you have the right IP and mac address)?
• ILO-information (address, credentials) (if you have to do cold migration)
• Has server been defragmented (this can significantly speed up conversion rates)?
• Has server been checked for hardware dongles?
• Has VLAN been trunked?
• Do server application licenses have any binding to MAC or IP address?

• Remote access type (RDP, Netop)? (for stopping services up front)
• Physical server location

• Applications on server
• What services to stop on server before conversion

• OS tester contact info
• Application tester contact info (for pre- and post migration test)

• Server to be converted by (employee)
• Date for conversion
• Conversion progress/status (not begun, P2V begun, handed over to OS testing, released to production, etc.)
• Has physical server been shut down?
• Notes

P2V post migration checklist

When doing P2V projects, I usually have a short, written checklist on my desk to make sure I remember everything. The list is as follows:

For hot migration:

1. Disable relevant services on the source machine
2. When configuring the P2V, don't set the VM for autoboot upon completion
3. Adjust hardware on the VM before first boot (remove serial ports etc)
4. Check that VMware Tools installs correctly
5. Adjust the HAL if needed
6. Uninstall HP software
7. Remove hidden NIC's
8. Set IP - if static IP (Start -> Run -> ncpa.cpl)
9. Check services.msc to ensure that all automatic services are running (and that you re-enabled the ones that you disabled to begin with)
10. Shutdown the physical server (shutdown /s /t 0 from CMD)
11. Ping -t the physical server and when it stops responding, then enable the NIC on the VM
12. Reboot the VM

After this, I typically handover the VM to the Windows Operations team which check the eventlog and such, and then they hand it over to the application testers before releasing it into production.

VLAN testing in ESX 3.5

In larger organisations, typically, the network department and the VMware group are seperated in different teams. So as a VMware administrator you need to ask the network department to trunk VLANs to the physical switch ports that your ESX is connected to. It happens that the network department misses a port or a VLAN which means that you can end up with a VM loosing network connection after e.g. a VMotion. Unfortunately, the responsibility can land on the VMware administrator for putting a host into production without testing VLAN connectivity. Unfair, but that's life.

But testing VLANs the manual way is rather time consuming. Especially if you have multiple hosts with multiple nics and multiple VLANs. The number of test cases quickly amount to the impossible. If, for example, you have five hosts, five VLANs and 4 NICs in each host, that means (5 x 5 x 4) 100 test cases.

The traditional way of testing is to create a vSwitch with only one vmnic connected. Then connect a VM on that vSwitch with one of the VLANs. Configure an IP address in the address space of the VLAN and ping the gateway. Do this for all the VLANs, and then connect the next vmnic to the vSwitch and start over.

The following method speeds up VLAN testing significantly (in this case from 100 to 16 test cases). It is not totally automated, but I have found it very useful nonetheless.

The basics of it is that you configure a port group to listen on all available VLANs and then you enable VLAN tagging inside the VM and do your testing from there:

1. Create a port group on the vSwitch with ID 4095. This will allow the VM to connect to all available VLANs available to the host.

2. Enable VLAN tagging from inside the VM. This only works with the E1000 intel driver which only ships with 64 bit Windows. So if you have a 32 bit Windows server, then you need to first modify the .vmx file and then download and install the intel E1000 driver from within Windows (Update: Even for Win 64 bit, you need to download and install E1000 manually. The advanced VLAN option is not included in the default driver). This link describes how this is done. Note that when modifying the .vmx, add the following line:

Ethernet0.virtualDev = "e1000"

Note that if you use the default Flexible nic to begin with, there's no existing entry for the nic in the .vmx, so just add the new entry.

Under Edit Settings for the VM, attach the NIC to the VLAN with id 4095.

3. Now you can add VLANs in the VM. Go to the Device Manager and then Properties for the E1000 NIC. There's a tab that says VLANs (see screendump below). As you add VLANs, a seperate NIC or "Local Area Connection" is created for each VLAN. It is set for DHCP, so if there's a DHCP server on that network it will receive an IP automatically. If not, you will need to configure an IP for that interface manually (e.g. by requesting a temporary IP from the network department.). For quickly configuring the IP, you can run the following command from CMD or a batch (.cmd) script:

netsh int ip set address "local area connection 1" static 192.168.1.100 255.255.255.0 192.168.1.254 1

4. Now we will use the Tracert (traceroute) command to test connectivity. The reason that we can't use Ping is the following: If you have multiple VLANs configured and you ping a gateway on a given VLAN - and the VLANs happen to be routable - then you will recieve a response from one of the other VLANs even though the one your are testing is not necessarily working.

But when using Tracert, then you can be sure that if the gateway is reached in the first jump, then the VLAN works. If the VLAN doesn't work, then you will see Tracert doing multiple jumps (via one of the other VLANs) before reaching the gateway (or it will fail if there's no connectivity at all). You can create a simple .cmd file with a list of gateways that you execute from the CMD prompt. Example file:

tracert 192.168.1.254

tracert 10.10.1.254

tracert 10.10.2.254

See below for example screendump.

Before running the batch script you need to have only one physical nic connected to the vSwitch. You can do this in one of two ways. 1) create a seperate vSwitch and connect only one vmnic at a time. Then you control it from VC. Or 2) you unlink all vmnics but one from the service console (COS) with the following commands:

ssh to the ESX host

esxcfg-vswitch -l (to see current configuration)

esxcfg-vswitch -U vmnic1 vSwitch0 (this unlinks vmnic1 from vSwitch0)

esxcfg-vswitch -L vmnic0 vSwitch0 (this links vmnic0 to vSwitch0)

These commands work instantaneously so you don't have to restart the network or anything. Then you run through the test on one vmnic at a time. When done with a host, you VMotion the VM to the next host in the cluster and continue the test from there.

Howto: Using Find command in Service Console

From time to time you need to locate stuff in the service console and the only command you got is find. 'Locate' unfortunately hasn't been included in the COS. Typically, I forget the syntax and think of another way of locating files - but actually it's pretty simple.

to use the Find command, do the following:

#find / -name searchstring

(#find 'path' -name 'searchstring')

so, if your looking for sshd_config file somewhere in /etc/ it would look like this:

#find /etc/ -name ssh_config

this will be a search on the complete file name. You can use wild cards as well, e.g.:

#find /etc/ -name ssh_co*

Prerequisites - Capacity Planner analysis

Before starting a Capacity Planner excercise, there is a number of things that should be in place. The following is typically what I send to customers and ask them to have in place beforehand:

No.	Prerequisite	Status
1.	Account with administrative rights on all servers to be surveyed, so: A Windows account(s) that have administrative privileges on local servers.	Username: Password: Domain:
2.	1 x Windows 2k3 or 2k8 server that we can install the Capacity Planner application on: Windows 2003 SP2 standard with min. 1 GB mem, 1 cpu 5 GB free on D-drive. Can be virtual. Should be joined to the domain where we collect data. We should have RDP-access to this server. This server can be virtual. The server needs internet access as performance data will be uploaded to optimize.vmware.com on port 80 and 443 TCP outbound.	Servername: Specifications: Domain: RDP available: Internet access:
3.	Local windows firewall should be disabled on clients to be surveyed or the following ports must be opened in local firewalls, inbound: TCP/UDP Ports: 135-139 and 445 (They are used for communication between Capacity Planner Data collector and windows hosts).	Firewalls disabled: Or ports opened:
4.	WMI and Remote Registry services should be running on all servers to be surveyed (typically they are running by default)	WMI is running: Remote Registry is running:
5.	A list of servers to be surveyed	List in .csv or .xls:

It is recommended that performance data is collected for a period of minimum 30 days and no less than 14 days.

Gartner strategic tech list 2010

In 2009, virtualization was way up on Gartner's list of strategic technologies. For 2010, virtualization again sets a big fingerprint on the list. Virtualization has been split up into several sub concepts which are represented indiviually on the list.