Configuring iSCSI for vSphere 5

Configuring a software iSCSI initiator for ESXi 5.0 is a relatively simple operation. This quick guide assumes that you have already configured an iSCSI target and published it on the network.

For inspiration, have a look at this VMware KB

Create a new vSwitch (Configuration -> Networking -> Add Networking) and add a VMkernel. Configure it with an IP address. 

Go to Storage adapters and click "Add" to add a software iSCSI adapter if it does not exist already.

Once added, right click the software initiator and choose "properties". 

Go to Network Configuration tab and click "Add".

Choose the vSwitch/VMkernel that you created above.

Go ot Dynamic Discovery tab and click "Add" to add an iSCSI target

You will be prompted to input IP address of the iSCSI target, just leave port 3260 as default unless you have configured it differently on your target.

Go to Configuration -> Storage and click "Add storage". Click DISK/LUN and next. If everything has been done correctly, you be able to see your published iSCSI target and can then add and format it with the new VMFS5 file system, uh lala!

vSphere 5 official release date

Update: 2011.08.25: vSphere has finally been released (as of 2011.08.24)! It can be downloaded from the VMware site.

Update 2011.08.23: Well, vSphere 5 was not released yesterday as rumors would have it - so I guess we'll just have to wait and see. A guess, not completely unreasonable, would be sometime during this week or on the first day of VMworld in the US...

When: The official release date of vSphere 5 - for GA - has been set to Monday 2011.08.22.

I found info about it here and I got the same info from our license vendor.

ESXTOP to the rescue - VM latency

Earlier on I have mostly used ESXTOP for basic troubleshooting reasons such as CPU ready and the like. Last weekend we had a major incident which was caused by a power outage which affected a whole server room. After the power was back on we had a number VMs that was showing very poor performance - as in it took about one hour to log in to Windows. It was quite random which VMs it was. The ESX hosts looked fine. After a bit of troubleshooting the only common denominator was that the slow VMs all resided on the same LUN. When I contacted the storage night duty the response was that there was no issue on the storage system.

I was quite sure that the issue was storage related but I needed some more data. The hosts were running v3.5 so troubleshooting towards storage is not easy.

I started ESXTOP to see if I could find some latency numbers. I found this excellent VMware KB article which pointed me in the right direction.

• For VM latency, start ESXTOP and press 'v' for VM storage related performance counters.
• The press 'f' to modify counters shown, then press 'h', 'i', and 'j' to toggle relevant counters (see screendump 2) - which in this case is latency stats (remember to stretch the window to see all counters)
• What I found was that all affected VMs had massive latency towards the storage system for DAVG/cmd (see screendump 1) of about 700 ms (rule of thumb is that max latency should be about 20 ms). Another important counter is KAVG/cmd which is time commands spend in the VMkernel, the ESX host, (see screendump 3). So there was no latency in the ESX host and long latency towards the storage system.

After pressing the storage guys for a while, they had HP come take a look at it, and it turned out that there was a defect fiber port in the storage system. After this was replaced everything worked fine and latency went back to nearly zero.

In this case, it was only because I had proper latency data from ESXTOP that I could be almost certain that the issue was storage related.

Screendump 1

Screendump 2

Screendump 3

Changing IP and VLAN on host - no VM downtime

It is possible to change the service console (COS) IP and VLAN id for hosts in a cluster without having VM downtime (see this post for changing hostname). The trick is to change the COS IP first on all hosts and then wait with the changing of the vMotion IP until all COS IP's have been changed. This way, you will be able to put the hosts into maintenance mode one by one and vMotion will still work with the old IP even though COS IP's will differ in range and VLAN id.

NB. It may be neccesary to disable HA for the cluster before you begin as the HA agent will not be able to configure on the hosts when IP's don't match for all hosts.

1. Enter maintenance mode
2. Update the DNS entry on the DNS server
3. Log on to the vCenter server and flush the DNS: ipconfig /flushdns
4. Go to ILO, DRAC or something similar for the host (you will loose remote network connection when changing the IP) and change the IP (use this KB article for inspiration): [root@server root]# esxcfg-vswif -i a.b.c.d -n w.x.y.z vswif0 , where a.b.c.d is the IP address and w.x.y.z is the subnet mask.
5. Change the VLAN id (in this case VLAN 12): esxcfg-vswitch -v 12 -p 'Service Console' vSwitch0
6. Change gateway: nano /etc/sysconfig/network
7. Change DNS servers: nano /etc/resolv.conf
8. Restart network: service network restart
9. Ensure that gateway can be pinged
10. Update the NTP server from the vSphere client if needed.
11. Continue the process with next host in the cluster

When all COS IP's have been changed, go to the vSphere client and change all vMotion IP addresses and VLAN id's. This will not require any downtime. And then test that vMotion works.

Done.

Changing hostname from the service console

The easiest way to change the hostname is via the vSphere client (see this post for changing IP address and VLAN IP). If, however, this is not an option for some reason, the hostname can be changed from the service console the following way:

This KB article actually explains most of the proces which includes:

-----------------

1. Open the /etc/hosts file with a text editor and modify it so that it reflects the correct hostname.

2. To change the default gateway address and the hostname, edit the /etc/sysconfig/network file and change the GATEWAY and HOSTNAME parameters to the proper values.

3. For the changes to take place, reboot the host or restart the network service with the command:

[root@server root]# service network restart
Note: This command breaks any current network connections to the Service Console, but virtual machines continue to have network connection.

------------------------------

I have experienced that after a reboot, the changes are reset and the hostname is changed back to the original one. To avoid this, there is one more step to be performed (before reboot):

Change the /adv/Misc/HostName parameter in /etc/vmware/esx.conf file (see screendump)

Error during upgrade: The system call API checksum doesn’t match

Today, I got an error during upgrade from vSphere 4.0 to 4.1 stating something like:

The system call API checksum doesn’t match

There was a lot of similar lines filling the console. I was a bit worried that the upgrade had gone wrong even though I had done three similar upgrades before this one with no errors - and that I would have to reinstall in stead.

Luckily, I found this error description in the 4.1 release notes stating that a reboot will fix the issue. So I waited for a while to be sure that the upgrade finished, rebooted, and everything looks fine:

Link to release notes:

"ESX service console displays error messages when upgrading from ESX 4.0 or ESX 4.1 to ESX 4.1 Update 1

When you upgrade from ESX 4.0 or ESX 4.1 release to ESX 4.1 Update 1, the service console might display error messages similar to the following:
On the ESX 4.0 host: Error during version check: The system call API checksum doesn’t match"
On the ESX 4.1 host: Vmkctl & VMkernel Mismatch,Signature mismatch between Vmkctl & Vmkernel

You can ignore the messages.

Workaround: Reboot the ESX 4.1 Update 1 host. "

Used network ports between ESX, vCenter, and the vSphere client

In a single customer setup, the network traffic and ports used in the virtual infrastructure are typically not a focus area because most components can be placed on the same networks. However, in a multiple customer environment we're experiencing that the network guys are asking a lot of questions as they want to lock down and secure the networks (which makes sense..).

Anyway, I just wanted to gather some of the info that I use regularly.

• A link to my previous post with a network diagram
• Link to VMware KB article about used ports for a vSphere environment and related components
• vMotion between networks requires TCP port 8000
• If searching and sorting VMs in the vSphere client is slow, then ensure that port 8443 TCP is opened between the vSphere client and the ESX hosts
• If hardware status tab is not available, then ensure that port 8443 TCP is opened between the vSphere client and the vCenter server
• In vCenter 5.x, if searching and sorting VMs in the vSphere client is slow then port 10443 TCP has to be opened between the vSphere client/client PC and the vCenter server (also, opening this port is required for viewing VM inventory across linked mode vCenter servers - for v5.1)
• If you can't get a remote console on the VMs (you get a black screen and a yellow bar in the top stating some sort of MKS error) ensure that port 903 (and 902) TCP is allowed from the vSphere client and to the ESX hosts
• If an ESX host keeps disconnecting in vCenter, ensure that port 902 UDP is allowed from the ESX host to the vCenter

Required ports between ESX and vCenter:

Source	Destination	Direction	Protocol	Port	Purpose
vCenter	ESX	In/out	TCP	902	VMware console
vCenter	ESX	In/out	TCP	903	VMware console
vCenter	ESX	In/out	TCP	443	HTTPS
vCenter	ESX	In/out	TCP	22	SSH
vCenter	ESX	In/out	TCP	80	HTTP
vCenter	ESX	In/out	TCP	161	SNMP
vCenter	ESX	In/out	TCP	5989	CIM
ESX	vCenter	out	UDP	902	Heartbeart