According to this VMware KB article there is no supported way to reset the root password on an ESXi v4 or v5 other than to reinstall it (or do a repair). I contacted VMware support and they sent me a guide for doing it in an unsupported way. This method is similar to what other guides on the web suggests. I went through the process but unfortunately i didn't work.
Finally, I mounted the ESXi 4.1 install ISO and did a repair. This resets most host configurations such a root password, network configuration, ntp settings, domain etc. After this I could set the password, reconnect to vCenter and then I had to reconfigure the host. Fortunately, the VMs were not completely gone from vCenter but were presented as greyed out orphaned VMs. So I could still see which LUNs the VMs were residing on. That way, the .vmx files could be located (except for one VM that had been renamed in vCenter without svMotioning or migrating it to another LUN afterwards...), the orphaned VM could be removed and the VM could be readded. It was quite a boring process but a least it worked.
Today, I wanted to recreate the password reset method in my home lab to see if I had actually done it in the correct way. I can confirm that, at least, on a virtual ESXi 5 it works and it is possible to reset the password to blank.
These are the steps (alternately, you can try this guide):
Download a Linux live bootable ISO. I used KNOPPIX. Mount the ISO and boot the host.
Once booted into KNOPPIX, open a shell.
Run the following set of commands:
# fdisk -l
# mkdir /mnt/disk
# mount /dev/sda5 /mnt/disk
(Mounting the correct device is the tricky part. To me, it was rather confusing which one to choose. For both the Fujitsu server that I dealt with and for the virtual ESXi, though, it was in sda5 that the state.tgz file was located. VMware suggested using the following command for HP servers: # mount /dev/cciss/c0d0p5 /mnt/disk - c0d0p5 is controller 0, disk 0, partition 5)
# cd /mnt/disk
# ls -al
# cp state.tgz state.tgz.bak
# cd /ramdisk
# mkdir temp
# cd temp
# tar zxf /mnt/disk/state.tgz
# ls -al
# tar zxf local.tgz
# cd etc
# nano shadow
Blank out the encrypted password. For example change root:$1$ywxtUqvn$9e1iXjGVd45T5IAgRxAuV.:13358:0:99999:7:::
See below screendumps for before and after:
Save the shadow file.
Run the following commands to repackage everything:
# cd ..
# rm -rf local.tgz
# tar zcf local.tgz *
# chmod 755 local.tgz
# rm -rf /mnt/disk/state.tgz
# tar zcf /mnt/disk/state.tgz local.tgz
# ls -al /mnt/disk/
# umount /mnt/disk
# shutdown -r now
When ESXi boots up it has no root password set (blank)