Practical commands for the ESXi console

For troubleshooting on ESX(i), I always end up in the console even on ESXi 5.x. There are a number of practical commands that I can usually remember, but not always. So here they are for reference in random order:

# tail -n 50 'file name'
Shows last 50 lines of a file

# tail -f 'file name'
Outputs continuously what is being written to the file

# more 'file name' | grep -C 10 'search string'
Outputs the line with the word you search for including 10 lines on each side of the entry.

# less 'file name'
A good alternative to 'more' and 'cat'. Lets you navigate back and forth with the keyboard arrows. Use 'w' for page up and 'z' for page down

# find / -name 'search string'
Search for something. Further described in this post.

# find -iname "*-flat.vmdk" -mtime +7 |  xargs ls –alh
Finds files older than 7 days and list them including when they've last been changed. See this post for more info on xargs.

Using the vi text editor. See this post.

Typen characters with ASCII code (hold the Alt key while inputting the number on the numeric keyboard). See more here.
@ - Alt+64
| - Alt+124

# esxcfg-scsidevs –a
# esxcli storage core adapter list
Both commands show info on the SCSI controller type, HBA type, WWNs. Here's more info.

# esxcfg-scsidevs –l
# esxcli storage core device list

Both commands show various info on LUNs including exact size

# esxcfg-mpath -l
# esxcli storage core path list
Shows info about the storage paths. Will show naa device id, LUN id, and state of the paths. You can grep for the word 'dead' for finding dead paths.

# dcui
Will show the yellow/grey ESXi console menu in a Putty session

# esxcli software vib list
# esxcli software vib update -v 'VIB file name'
Updates the VIB package. See here for more info

# pwd
Show working directory

# passwd
Change password for current user (can be used for root as well)

# date
Show date and time.

False CIM warning on the SCSI controller - vSphere 5.

We have had a number of CIM warnings in the Hardware Status tab in vCenter on the ESXi 5 hosts related to the SCSI controller. We have seen it on HP servers with a P220i and a P410i SCSI controller.

However, there is nothing wrong with the SCSI controller.

Turns out it is a false alarm generated by the hp-smx-provider and it can be fixed by applying a driver update (which effectively changes the hp-smx-provider with the hp-smx-limited driver, both are CIM drivers). The difference between the two is, to my understanding, that with the limited driver you cannot update the ILO firmware via the ESXi console.

There are two ways of installing the four VIBs in the driver package. Either you can uninstall the existing four VIBs and then install the driver package, the zip file, which will install all four VIBs. This requires two reboots.

Or you can unpack the zip file and update only the VIBs that are actually newer than the ones you already have installed. In our case, we only needed to update two of the VIBs. This requires one reboot.

Method 1 (uninstall/install)

List the relevant VIBs

# esxcli software vib list | grep hp

Remove the installed VIBs

# esxcli software vib remove -n hp-smx-provider
# esxcli software vib remove -n char-hpcru
# esxcli software vib remove -n char-hpilo
# esxcli software vib remove -n hp-ams

Reboot

Download the driver package fromt the HP Support Center. The file is called "hp-ams-esxi5.0-bundle-9.3.5-3.zip".

Upload it to the ESXi host and place it in /tmp/.

# cd /tmp/

# esxcli software vib install -d /tmp/hp-ams-esxi5.0-bundle-9.3.5-3.zip

Reboot

Verify that the new VIBs have been installed

# esxcli software vib list | grep hp

Method 2 (update)

List the relevant VIBs

# esxcli software vib list | grep hp

Unpack the "hp-ams-esxi5.0-bundle-9.3.5-3.zip" file and copy the following VIB files to the /var/log/vmware folder:

• hp-ams-esx-500.9.3.5-02.434156.vib
• hp-smx-limited-500.03.02.10.3-434156.vib

If you unsure about whether the other two VIBs are newer than the ones already installed, then just copy all four VIBs.

Update the VIBs

# esxcli software vib update -v hp-ams-esx-500.9.3.5-02.434156.vib

# esxcli software vib update -v hp-smx-limited-500.03.02.10.3-434156.vib

Reboot

Verify that the new VIBs have been installed

# esxcli software vib list | grep hp

When the host comes back online after the reboot it will still have the warning. Leave it for about 10-15 minutes and the warning will disappear. Also, you can try to reset the sensors and refresh the page.

Thanks to Anders Mikkelsen for finding this fix and writing the guide to method 1 above.

Updating the firmware and driver on an HBA on ESXi 5.0

When updating firmware on an HBA, it will update the driver at the same time as both parts are included in the installation VIB file.

First step is to verify the type of HBA in your ESXi host:

Run the following command:

# esxcfg-scsidevs -a

Or see this KB article for more info

In the example below, the HBA is a Qlogic ISP2532 PCI-Express

However, this name will not always be used to (at least not in the case of HP) in the documentation when you're looking for a specific model.

Another way to get info on the HBA type can be combined with locating the firware version (see next step)

Next step is to identify the current firmware level of the HBA:

For a Qlogic HBA, use the following command:

# more /proc/scsi/qla2xxx/0

(The '0' can also be a '1' or a '2')

For other types of HBAs, see this KB article

In the screendump above, you can see firmware and driver version and also another name for the HBA, QMH2562.

Now that we know the model and firmware version, we can look for the latest recommended version.

Go to this KB article to find the latest version.

For HP, it takes you to the following site: http://vibsdepot.hp.com/hpq/recipes/ where you should choose the PDF called: "HP-VMware-Recipe.pdf" which is the latest one.

In the PDF, you can search for you model, in this case QMH2562:

Find your model and follow the link. It will let you download a zip file. In this example the file is called:

qla2xxx-934.5.6.0-887798.zip

Unpack the zip file and locate the .vib file which will be used for the update.

In this case the .vib file is called:

scsi-qla2xxx-934.5.6.0-1OEM.500.0.0.472560.x86_64.vib

Now, transfer the .vib file to the following folder on the ESXi host:

/var/log/vmware

Run the following command:

# esxcli software vib update -v scsi-qla2xxx-934.5.6.0-1OEM.500.0.0.472560.x86_64.vib

Reboot to finish updating.

And then verify that the update has properly installed by re-running:

# more /proc/scsi/qla2xxx/0

You can also get detailed VIB package info by running:

# esxcli software vib list | grep scsi
# esxcli software vib get -n scsi-qla2xxx

Locating orphaned vmdk's

When you have had a VMware environment running for several years and have had many admins interacting with this environment, there is a fair chance that there will be a number of orphaned vmdk's on the LUNs which are taking up valuable space.

I found a post on yellow-bricks.com explaining two different ways of locating these vmdk's:

The first option is to simply look for all flat files that haven't been changed during the past seven days. This is easy and it works on ESXi 5.0. However, be careful before you start deleting files as a powered off VM, for example, also will show up in such a search (as well as probably also VMs with snapshots).

# find -iname "*-flat.vmdk" -mtime +7

Alternatively, you can pipe the output of the find command to ls see when each file has been modifed:

# find -iname "*-flat.vmdk" -mtime +7 | xargs ls –alh

And if you feel very safe that all the files can be deleted, you delete them all at once:

# find -iname "*-flat.vmdk" -mtime +7 | xargs rm -f

The second option which is a more thorough way of doing it is to run a powershell script (can also be found in the post) which looks for vmdk's which are not registered to a .vmx file. The script has been slightly modified by Danni Finne and can be found here:

ESXi 5.0 host disconnects due to CPU crossdup error

If a host is disconnected in vCenter and you cannot reconnect the host and you see the following entries in the /var/log/vmkwarning file:

2013-07-05T06:54:02.084Z cpu7:7757089)WARNING: UserObj: 675: Failed to crossdup fd 13, /vmfs/devices/char/vob/External type CHAR: Busy

2013-07-05T06:54:02.084Z cpu7:7757089)WARNING: UserObj: 675: Failed to crossdup fd 14, /vmfs/devices/char/vob/iScsi type CHAR: Busy

2013-07-05T06:54:02.084Z cpu7:7757089)WARNING: UserObj: 675: Failed to crossdup fd 15, /vmfs/devices/char/vob/Migrate type CHAR: Busy

2013-07-05T06:54:02.084Z cpu7:7757089)WARNING: UserObj: 675: Failed to crossdup fd 16, /vmfs/devices/char/vob/PageReti type CHAR: Busy

2013-07-05T06:54:02.084Z cpu7:7757089)WARNING: UserObj: 675: Failed to crossdup fd 17, /vmfs/devices/char/vob/Visorfs type CHAR: Busy

2013-07-05T06:54:02.084Z cpu7:7757089)WARNING: UserObj: 675: Failed to crossdup fd 18, /vmfs/devices/char/vob/Hardware type CHAR: Busy

2013-07-05T06:54:02.084Z cpu7:7757089)WARNING: UserObj: 675: Failed to crossdup fd 19, /vmfs/devices/char/vob/Vfat type CHAR: Busy

2013-07-05T06:54:02.084Z cpu7:7757089)WARNING: UserObj: 3232: Unimplemented operation on 0x410028907210/SOCKET_UNIX_SERVER

It can be fixed by following this KB:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2031894

So, SSH to the host and edit the file (/etc/vmware/vpxa/vpxa.cfg) using vi text editor.

The following line should be changed, the value should be increased to 256 from originally 128:

'ThreadStackSizeKb'256'/ThreadStackSizeKb'

Vpxa.cfg is read-only so you can change it with:

# chmod 744 vpxa.cfg

And when done – to get file back to read only:

# chmod 444 vpxa.cfg

Restart the vpxa service and reconnect the host.

# /etc/init.d/vpxa restart

Manually recreating a vmdk descriptor file

Yesterday, I had an svMotion that crashed as the vCenter server (vSphere 5.0 u1) was rebooted while the svMotion task was in progress. Afterwards, a copy of the VM named "servername (2)" had been created in vCenter and a number of empty vmdk files were created at the destination LUNs. There were a file lock on these files so they could not be deleted. and I wasn't able to vMotion, snapshot or continue the svMotion on the source VM.

I tried moving just one of the vmdks with svMotion which resulted in the VM crashing and after that it wouldn't power back on.

Below is the error:

The error states that it cannot find one of the vmdk's. I looked in the relevant folder and discovered that the vmdk descriptor file was missing, the folder only contained the vmdk-flat file (the data file).

Under Edit Settings for the VM, the disk was still there but its size was set to 0 MB, see below:

I found a KB from VMware on how to recreate this descriptor file. And it worked fine.

Loosely, the steps were the following (the KB explains it in details):

Identify scsi controller type, lsilogic in this case.

Identify the size of the vmdk-flat file

Create the vmdk as a thin vmdk

Delete the empty vmdk data file

Edit the vmdk descriptor file with the vi text editor. This included the vmdk flat file name, the vHW version, and deleting the line about thin provisioning.

Verify the consistency of the descriptor with vmkfstools -e:

Locate WWN from console on ESXi 5.x

Sometimes for urgent cases it can be necessary to obtain the HBA's WWN's to get the storage zoned before the network configurations are done and the hosts are online. On Blade servers, this info can be found on the enclosure OA but for rack mounted servers you can only get it from the console.

From the console (press Alt-F1 at the console, remember to enable shell access first under troubleshooting) login as root and run the following command:

# esxcfg-scsidevs -a

look for the lines starting with vmhba1 and vmhba2 (vmhba0 is typically the scsi controller) and the fc.XXX:XXX. The last numbers after the ":" is the WWN (see screen dump below)

Basic vi text editor commands

As the Nano editor unfortunately has been taken out of the ESXi 5.0 shell, we're left with good old vi.

Here's some basic commands:

Opening a text file:

vi filename

vi opens in Command mode. You can move the cursor around in the file but not edit it. To switch edit mode press 'i' (you can see that a '-' symbol at the bottom of the console changes to an 'I'). To switch back to Command mode press 'Esc'.

To save:

:w

To quit:

:q!

See this link for more info

View HBA firmware version from service console

To view HBA firmware version from service console (ESX classic) go to /proc/scsi/qla or lpfc820.
Here you will typically find to text files, e.g. '2' and '3'. Run a 'cat' or 'more' on the files (see screendump below). See this post for more info

For ESXi v5.x, see this link.

Large VM crashes during snapshot commit

Snapshots can be your friend but they can most certainly also make your life miserable. The other day we had a rather large VM (with 20 GB mem, 8 vCPUs and 28 TB storage divided on 22 .vmdk's) that crashed during a snapshot commit. The error stated: "Performing disk cleanup. Cannot power off." The snapshot had been taken while the VM was powered off and only a few changes had been made to the VM before the snapshot was committed.


After the crash, the VM would not power on. The error stated: "Reason: Cannot allocate memory" and in the error  description (see screendump below) there's an indication of disk a lock or disk error. Fortunately, the VM could be started from the service console (ESX 4.1 classic) with 'vmware-cmd'.

After boot, vCenter stated that there was no snapshots on the VM. However, 22 delta files on a single LUN was telling otherwise.

A normal procedure to do cleanup is to power off VM and clone it. However, with 28 TB storage in the VM, this was not an option.

Instead, the following did the trick: Log on to the service console, change directory to the folder where the .vmx file for the VM resides, take a new snapshot and then do a remove all snapshots (see this KB article for more info). This removes the new snapshot as well as the 'defect' snapshot.

To see if any snapshots exist (that will probably not be the case): 

vmware-cmd vmname.vmx hassnapshot

To take new snapshot (with no quiesce and no memory, see this KB article for details)

vmware-cmd vmname.vmx createsnapshot snapshot-name description 0 0

As you can see in screen dump below at first I tried to run the command without the two boolean arguments that relates to QuiesceFilesystem and IncludeMemory. 

To remove all snapshots:

 vmware-cmd vmname.vmx removesnapshots

In the screendump above the removesnapshots command returns an error code '1' which means that all is well and snapshots are gone.

Could not power on VM - lock was not free

The other day we experienced an incident on the SAN storage with high latency and even loss of connection to the SAN. This can generate a lot of really unpleasant errors on the ESX hosts. Even after the SAN is brought back to a stable state we've seen hosts that won't boot, VM's that won't vMotion and VMs that won't power on due to file locks. 

If you receive a 'locked file error' (like screendump below) and your VM won't boot there are a couple of ways to go about it. This VMware KB article explains it quite well. Either you can cold migrate the VM to the  other hosts in the cluster (to find the ESX host with the lock) and then try to boot it from there or you can try to locate specifically which host has the lock.

If the vCenter log does not tell you specifically which files are locked, this can be viewed in the vmware.log which is located in the VM folder. If you just tried to power on the VM, then relevant info should be at the end of the log file.

In the example below, it is the swap that is still locked.

This can be verified by running the touch command on the locked file.
With vmkfstools you can get the mac address that has the lock:

# vmkfstools -D /vmfs/volumes///

In the screendump below, the MAC address has been highlighted.

The same info can be found in the /var/log/vmkernel log

Once you have the MAC address you can find a match by, for example, logging in to vCenter or onto the Blade enclosure. When you have a match, cold migrate the VM to the relavant ESX host and boot it.

ESXTOP to the rescue - VM latency

Earlier on I have mostly used ESXTOP for basic troubleshooting reasons such as CPU ready and the like. Last weekend we had a major incident which was caused by a power outage which affected a whole server room. After the power was back on we had a number VMs that was showing very poor performance - as in it took about one hour to log in to Windows. It was quite random which VMs it was. The ESX hosts looked fine. After a bit of troubleshooting the only common denominator was that the slow VMs all resided on the same LUN. When I contacted the storage night duty the response was that there was no issue on the storage system.

I was quite sure that the issue was storage related but I needed some more data. The hosts were running v3.5 so troubleshooting towards storage is not easy.

I started ESXTOP to see if I could find some latency numbers. I found this excellent VMware KB article which pointed me in the right direction.

• For VM latency, start ESXTOP and press 'v' for VM storage related performance counters.
• The press 'f' to modify counters shown, then press 'h', 'i', and 'j' to toggle relevant counters (see screendump 2) - which in this case is latency stats (remember to stretch the window to see all counters)
• What I found was that all affected VMs had massive latency towards the storage system for DAVG/cmd (see screendump 1) of about 700 ms (rule of thumb is that max latency should be about 20 ms). Another important counter is KAVG/cmd which is time commands spend in the VMkernel, the ESX host, (see screendump 3). So there was no latency in the ESX host and long latency towards the storage system.

After pressing the storage guys for a while, they had HP come take a look at it, and it turned out that there was a defect fiber port in the storage system. After this was replaced everything worked fine and latency went back to nearly zero.

In this case, it was only because I had proper latency data from ESXTOP that I could be almost certain that the issue was storage related.

Screendump 1

Screendump 2

Screendump 3

Changing IP and VLAN on host - no VM downtime

It is possible to change the service console (COS) IP and VLAN id for hosts in a cluster without having VM downtime (see this post for changing hostname). The trick is to change the COS IP first on all hosts and then wait with the changing of the vMotion IP until all COS IP's have been changed. This way, you will be able to put the hosts into maintenance mode one by one and vMotion will still work with the old IP even though COS IP's will differ in range and VLAN id.

NB. It may be neccesary to disable HA for the cluster before you begin as the HA agent will not be able to configure on the hosts when IP's don't match for all hosts.

1. Enter maintenance mode
2. Update the DNS entry on the DNS server
3. Log on to the vCenter server and flush the DNS: ipconfig /flushdns
4. Go to ILO, DRAC or something similar for the host (you will loose remote network connection when changing the IP) and change the IP (use this KB article for inspiration): [root@server root]# esxcfg-vswif -i a.b.c.d -n w.x.y.z vswif0 , where a.b.c.d is the IP address and w.x.y.z is the subnet mask.
5. Change the VLAN id (in this case VLAN 12): esxcfg-vswitch -v 12 -p 'Service Console' vSwitch0
6. Change gateway: nano /etc/sysconfig/network
7. Change DNS servers: nano /etc/resolv.conf
8. Restart network: service network restart
9. Ensure that gateway can be pinged
10. Update the NTP server from the vSphere client if needed.
11. Continue the process with next host in the cluster

When all COS IP's have been changed, go to the vSphere client and change all vMotion IP addresses and VLAN id's. This will not require any downtime. And then test that vMotion works.

Done.

Changing hostname from the service console

The easiest way to change the hostname is via the vSphere client (see this post for changing IP address and VLAN IP). If, however, this is not an option for some reason, the hostname can be changed from the service console the following way:

This KB article actually explains most of the proces which includes:

-----------------

1. Open the /etc/hosts file with a text editor and modify it so that it reflects the correct hostname.

2. To change the default gateway address and the hostname, edit the /etc/sysconfig/network file and change the GATEWAY and HOSTNAME parameters to the proper values.

3. For the changes to take place, reboot the host or restart the network service with the command:

[root@server root]# service network restart
Note: This command breaks any current network connections to the Service Console, but virtual machines continue to have network connection.

------------------------------

I have experienced that after a reboot, the changes are reset and the hostname is changed back to the original one. To avoid this, there is one more step to be performed (before reboot):

Change the /adv/Misc/HostName parameter in /etc/vmware/esx.conf file (see screendump)

Restart of ESX management agents

This is just a post to remember the commands for restarting the management agents on an VMware ESX server:

#service mgmt-vmware restart

#service vmware-vpxa restart (the HA agent)

Both of these agents can be restarted without affecting VM operation. Restarting them can be a useful step in troubleshooting if vCenter has trouble connecting to a host or if you experience HA errors.

Here's link to KB article

For restarting mgmt agents in ESXi, this can be done via the console menu interface, see link above.

Console-setup - service console tool for network config in ESX4

As of ESX 4.0 U2 a new tool for configuring network in the service console (COS) has been introduced. If you're not too comfortable with CLI then this might come in handy. The tool will give you a numbered menu and you can list and configure nics, vswitches, vswifs, etc.

Here's a link to a VMware KB article that presents the tool.

To run the tool, type console-setup in the COS.

Menu entry 1, 2, and 3 will show the output of esxcfg-vswif -l, esxcfg-nics -l, and esxcfg-vswitch -l respectively.

Menu entry 5 will let you configure your service console without having to remember any of the commands. Pretty neat..

Here's a link on how to do it the old school way.

Example of an HA error - and a fix

The other day, I got an HA error when trying to add a new host into a cluster. It was weird, as the host was identical to the others - same model, same installation procedure, and everything. In VirtualCenter, the error looked like this:

This piece of information did not help much in relation to troubleshooting.

The only thing that was different with the new host was that is was configured from the service console (COS) as its NICs were DOA. I had used my own guide for this, so I thought I was in good shape ;-).

A more descriptive error was to be found in the VirtualCenter agent log file on the host (/var/log/vmware/vpx/vpxa.log). Grepping for the word "error" gave the following output:

errorcat = "hostipaddrsdiffer",

errotext = "cmd addnote failed for primary node: Host misconfigured. IP address of ... not found on local interface"

Earlier on, I had changed the IP address, as the first one assigned was already in use, but I'd forgotten to change the IP address in the /etc/hosts file. After doing that and restarting the network (service network restart), everything worked fine.

As a side node, I can mention that it can be pretty confusing manoeuvering through the various log files. Check this post by Eric Siebert for further explanation of VMware log files on VI3.

VLAN testing in ESX 3.5

In larger organisations, typically, the network department and the VMware group are seperated in different teams. So as a VMware administrator you need to ask the network department to trunk VLANs to the physical switch ports that your ESX is connected to. It happens that the network department misses a port or a VLAN which means that you can end up with a VM loosing network connection after e.g. a VMotion. Unfortunately, the responsibility can land on the VMware administrator for putting a host into production without testing VLAN connectivity. Unfair, but that's life.

But testing VLANs the manual way is rather time consuming. Especially if you have multiple hosts with multiple nics and multiple VLANs. The number of test cases quickly amount to the impossible. If, for example, you have five hosts, five VLANs and 4 NICs in each host, that means (5 x 5 x 4) 100 test cases.

The traditional way of testing is to create a vSwitch with only one vmnic connected. Then connect a VM on that vSwitch with one of the VLANs. Configure an IP address in the address space of the VLAN and ping the gateway. Do this for all the VLANs, and then connect the next vmnic to the vSwitch and start over.

The following method speeds up VLAN testing significantly (in this case from 100 to 16 test cases). It is not totally automated, but I have found it very useful nonetheless.

The basics of it is that you configure a port group to listen on all available VLANs and then you enable VLAN tagging inside the VM and do your testing from there:

1. Create a port group on the vSwitch with ID 4095. This will allow the VM to connect to all available VLANs available to the host.

2. Enable VLAN tagging from inside the VM. This only works with the E1000 intel driver which only ships with 64 bit Windows. So if you have a 32 bit Windows server, then you need to first modify the .vmx file and then download and install the intel E1000 driver from within Windows (Update: Even for Win 64 bit, you need to download and install E1000 manually. The advanced VLAN option is not included in the default driver). This link describes how this is done. Note that when modifying the .vmx, add the following line:

Ethernet0.virtualDev = "e1000"

Note that if you use the default Flexible nic to begin with, there's no existing entry for the nic in the .vmx, so just add the new entry.

Under Edit Settings for the VM, attach the NIC to the VLAN with id 4095.

3. Now you can add VLANs in the VM. Go to the Device Manager and then Properties for the E1000 NIC. There's a tab that says VLANs (see screendump below). As you add VLANs, a seperate NIC or "Local Area Connection" is created for each VLAN. It is set for DHCP, so if there's a DHCP server on that network it will receive an IP automatically. If not, you will need to configure an IP for that interface manually (e.g. by requesting a temporary IP from the network department.). For quickly configuring the IP, you can run the following command from CMD or a batch (.cmd) script:

netsh int ip set address "local area connection 1" static 192.168.1.100 255.255.255.0 192.168.1.254 1

4. Now we will use the Tracert (traceroute) command to test connectivity. The reason that we can't use Ping is the following: If you have multiple VLANs configured and you ping a gateway on a given VLAN - and the VLANs happen to be routable - then you will recieve a response from one of the other VLANs even though the one your are testing is not necessarily working.

But when using Tracert, then you can be sure that if the gateway is reached in the first jump, then the VLAN works. If the VLAN doesn't work, then you will see Tracert doing multiple jumps (via one of the other VLANs) before reaching the gateway (or it will fail if there's no connectivity at all). You can create a simple .cmd file with a list of gateways that you execute from the CMD prompt. Example file:

tracert 192.168.1.254

tracert 10.10.1.254

tracert 10.10.2.254

See below for example screendump.

Before running the batch script you need to have only one physical nic connected to the vSwitch. You can do this in one of two ways. 1) create a seperate vSwitch and connect only one vmnic at a time. Then you control it from VC. Or 2) you unlink all vmnics but one from the service console (COS) with the following commands:

ssh to the ESX host

esxcfg-vswitch -l (to see current configuration)

esxcfg-vswitch -U vmnic1 vSwitch0 (this unlinks vmnic1 from vSwitch0)

esxcfg-vswitch -L vmnic0 vSwitch0 (this links vmnic0 to vSwitch0)

These commands work instantaneously so you don't have to restart the network or anything. Then you run through the test on one vmnic at a time. When done with a host, you VMotion the VM to the next host in the cluster and continue the test from there.

Howto: Using Find command in Service Console

From time to time you need to locate stuff in the service console and the only command you got is find. 'Locate' unfortunately hasn't been included in the COS. Typically, I forget the syntax and think of another way of locating files - but actually it's pretty simple.

to use the Find command, do the following:

#find / -name searchstring

(#find 'path' -name 'searchstring')

so, if your looking for sshd_config file somewhere in /etc/ it would look like this:

#find /etc/ -name ssh_config

this will be a search on the complete file name. You can use wild cards as well, e.g.:

#find /etc/ -name ssh_co*

Howto: Check if SAN cables are connected in ESX

When installing an ESX host and you have someone other than yourself taking care of the cabling of the host, it is very handy to be able to check wheather this has been done properly. You want to be able to verify that the HBA's have been physically connected to the fabric switches with fibre cables.

Ssh to the ESX host

ls to the /proc/scsi/qla2300 folder (if it's a Qlogic HBA...)

In this folder there are a number of text files named with the numbers 1-x corresponding to the number of HBA ports in your ESX.

Cat the files one at a time:

#cat 1

or

#cat /proc/scsi/qla2300/1

look for the following line in the files:

Host adapter:loop state=READY, flags= 0x8430403

If it says READY, the HBA has been physically connected to the fibre switch. If it says DEAD, then it is not.