P2V error - BlockLevelVolumeCloneMgr and Sysimgbase_DiskLib_Write

The other day we had to do a number of hot P2V's on some Citrix servers running Win2k3. I had succesfully completed a test migration a week before (with VMware Converter Standalone 5 installed locally on the source) on one of the same servers but when we re-initiated the P2V in the planned maintenance window, both servers failed at 90-something percent with an error stating the following:

SingleVolumeCloneTask:DoRun: Volume cloning failed with clone error BlockLevelVolumeCloneMgr::CloneVolume: Detected a write error during the cloning of volume \WindowsBitmapDriverVolumeId=[08-03-AE-BE-00-40-00-00-00-00-00-00]. Error: 37409 (type: 1, code: 2338)

This log entry is found by right clicking the job in Converter and choosing 'export logs'. Locate the file called vmware-converter-worker-X.log (where X is an incremental integer).

The above error message seems to indicate that there is a problem on the source disk. We tried running checkdisk which showed no errors and we defragmented all drives. Same error occurred.

Looking a bit more at the logs, I found the following entries which pointed towards a network error:


[NFC ERROR] NfcSendMessage: send failed: NFC_NETWORK_ERROR
[NFC ERROR] NfcFssrvr_IO: failed to send io message
Sysimgbase_DiskLib_Write failed with 'NBD_ERR_NETWORK_CONNECT' (error code:2338)

By searching a bit on the above entries, I was pointed towards a relevant KB article from VMware. As it turns out, this is not network related at all, it is a known error in the Converter Standalone (both v4 and v5) software. The KB simply states that VMware is aware of this issue... I've done a ton of P2V's but this error I've never seen before...

The good news is that there is a workaround:

The trick is to only transfer one drive at a time. This means that if the source has a C and a D drive you'll be P2V'ing this machine twice creating to seperate VMs - one only containing the C drive including the system partition and another VM (which I just called 'servername_Ddrive') containing only the D drive. When both P2V's are done the second one is removed from inventory. For the first VM, go to Edit Settings and attach the disk from the second VM, 'servername_Ddrive'. After that, you can boot the VM now containing both drives. Be aware that the newly attached disk will deafult to drive letter D. This means that if it had another drive letter before, you'll have to change it manually.

An important point to mention in this process is that when transferring the second VM only containing the D drive, the transfer will fail with an error around 98% stating something like "An error ocurred during reconfiguration...". This is ok - as long as the drive has been succesfully cloned, this is what matters (see below).

An alternative workaround that will most likely work as well is to do a cold clone.

Below is a screen dump of the releant entries in the log file.

P2V with VMware Converter Standalone 5 and sync feature

For this blog post number 100 (uh la laa) I decided to spice things up a bit with video tutorial showing some of the new features in VMware vCenter Converter Standalone 5 including the enhanced synchronize feature (it also existed in v4 but it didn't work too well..).

The video guides you through the migration wizard and discusses some relevant use cases for the sync feature.

Here's a link to Converter Standalone 5

Remember that as of v4.3 Windows Server 2000 is no longer supported as a source OS, so to convert win2k use Converter Standalone v4.01 in stead. In the release notes you can see supported guest operating systems.

P2V pre-migration checklist - and considerations

My prevoius post was a P2V post migration checklist. This post is a pre-migration checklist which is about all the information that should gathered and checked before doing any P2V conversions.

I have been involved in a number of larger P2V projects (+50 P2V's) and, in my experience, proper planning is a key element for a succesful project. Typically, you, as a VMware- or P2V person, have no real knowledge of the Windows servers to be converted - their just another server. This means that you rely on other people to collect relevant data on your behalf. Such a setup has an important implication. As you have no knowledge of the server, it cannot be released into production by yourself, you should let a Windows guy verify the OS after which it can be handed over for application testing. Resources for both tests should be allocated up front by the project manager and they should be standing by in the agreed maintenance window.

In regards to the length of maintenance windows, we have had the most succes with long time frames during weekends - e.g. 36 hours from Saturday 08.00 a.m. to Sunday 08.00 p.m. Obviously, such a window can be difficult to obtain, but it has two significant advantages: 1) Specifying the actual conversion time can be tricky - it happens that a 30 GB server takes 12 hours to convert for one reason or the other. 2) It is less stressfull to do P2V's during weekends and a long window will let you work at your own pace, Furthermore, conversions can run over night if they have large disks (e.g. + 200 GB).

Now, a few words about the checklist. Over time, it has been gradually extended as we have learned important lessons - some of them the hard way where. For example, a server that hadn't been checked for hardware dongles, then you need to roll back - or e.g. a VLAN that hadn't been properly trunked... A specific list will match a specific scenario so, typically, the list will be modified to some degree for each project. However, a large part of the list will remain the same, so hopefully it can be used for inspiration. We use Sharepoint 2007 to organise the lists. These can be dynamically updated, which is practical when multiple persons have to update at the same time.

• Servername
• OS type
• Server model

• Has Capacity Planner run for this server?

• # of CPU sockets
• # of CPU cores
• Amount of physical memory installed
• Physical disk capacity (C-drive, D-drive, etc.)

• Current CPU usage (preferably from cap. planner)
• Current memory usage (preferably from cap. planner)
• Current physical disk usage (C-drive, D-drive, etc.)

• # vCPU’s that should be assigned
• Amount of memory to be assigned to VM
• Sizes of vDisks after resizing (C-drive, D-drive, etc. – remember separate .vmdk’s for each logical volume)
• Total size of vDisks (then you can sum up total disk capacity needed and ask for storage up front)

• Local administrator credentials (local windows accounts are recommended)
• “Ipconfig /all” screendump attached to list (this is to ensure you have the right IP and mac address)?
• ILO-information (address, credentials) (if you have to do cold migration)
• Has server been defragmented (this can significantly speed up conversion rates)?
• Has server been checked for hardware dongles?
• Has VLAN been trunked?
• Do server application licenses have any binding to MAC or IP address?

• Remote access type (RDP, Netop)? (for stopping services up front)
• Physical server location

• Applications on server
• What services to stop on server before conversion

• OS tester contact info
• Application tester contact info (for pre- and post migration test)

• Server to be converted by (employee)
• Date for conversion
• Conversion progress/status (not begun, P2V begun, handed over to OS testing, released to production, etc.)
• Has physical server been shut down?
• Notes

P2V post migration checklist

When doing P2V projects, I usually have a short, written checklist on my desk to make sure I remember everything. The list is as follows:

For hot migration:

1. Disable relevant services on the source machine
2. When configuring the P2V, don't set the VM for autoboot upon completion
3. Adjust hardware on the VM before first boot (remove serial ports etc)
4. Check that VMware Tools installs correctly
5. Adjust the HAL if needed
6. Uninstall HP software
7. Remove hidden NIC's
8. Set IP - if static IP (Start -> Run -> ncpa.cpl)
9. Check services.msc to ensure that all automatic services are running (and that you re-enabled the ones that you disabled to begin with)
10. Shutdown the physical server (shutdown /s /t 0 from CMD)
11. Ping -t the physical server and when it stops responding, then enable the NIC on the VM
12. Reboot the VM

After this, I typically handover the VM to the Windows Operations team which check the eventlog and such, and then they hand it over to the application testers before releasing it into production.

Supported image formats for Converter 4 Standalone

VMware Converter 4 Standalone supports the following image formats (see screen dump below). Hyper-V is supported but conversion will have to be done as a physical server (see link)

http://www.vmware.com/products/converter/faqs.html

vCenter Converter Standalone 4 - ports used

We're doing quite a few P2V conversions at the moment, and that means that we see all kinds of weird errors, conversion failures, and connection issues. P2V is definitely not an exact science.

One thing that is recommended to have in order is that proper network ports are opened.

VMware has written a good KB article that explains which ports are used.

If you have server with Converter Standalone installed on it, and you have trouble connecting to the source physical computer, then first make sure that Windows Firewall is disabled. If that doesn't work, then install the Converter application directly on the source computer. Then you will need outbound 443 TCP connection to vCenter (former Virtual Center) (it's assumed that port 443 TCP is open inbound on the vCenter server, of course).

To test if ports are open, open a CMD prompt and run following command:

telnet 'vCenter ip' 443

(without the ' ') If the DOS prompt goes black, then the connection is good. Othervise you will get a 'can't connect' or something similar)

If you P2V directly to an ESX server, then ports 902, 903, and 443 TCP are used.

If you, for some reason, can't get port 443 opened, then a workaround is as follows:

• Install the Converter directly on the source system
• If you have an existing test VM in the same IP range, then create a new disk and attach that to the test VM.
• Make a Windows share on the new disk
• From the Converter choose to export to standalone virtual machine in Workstation format and then coose to place files on the share just created
• After export, change the VLAN to an IP range that doesn't have any firewalls blocking
• Import the VM from within vCenter

P2V of domain controller

Summary: Cold clone P2V of domain controllers works just fine.


We had to migrate two root domain controllers the other day at work. I knew that domain controllers in particular can give you trouble when being converted / migrated, so I researched it a bit and found a useful article on yellow-bricks.com which linked to a very good VMware KB article . This KB recommends that in stead of migrating, then deploy a fresh VM and do a 'dcpromo' and then shut down the physical server after. I like this way as it moves the responsibility away from the VMware team and over to the application responsible.

However, we did not have enough time to do the recommended solution, so we whent for P2V. We did cold clone because hot migration is likely to go wrong and it is not supported by Microsoft.


There were FSMO roles on the DC's, so before we began, we had the AD guy move all the roles over to one of the servers. Then we took the other one down and P2V'ed it. We resized the disks to save SAN space which was not a problem. When it came back up, the AD guy tested and then moved FSMO roles over to the migrated DC. And then we migrated the other one. After both had been migrated, the AD guy tested again.


If your responisbility area does not cover the application layer, which it does not for me in this case, then arrange for an application responisble to test the app before it is released into production. It may sound banal, but it is sometimes overlooked when the pace is fast and only basic OS testing is done.


Time synchronization


There are several ways of setting up time synchronization. One important point is that there should be only one source for synchronization for all the DC's. There's a feature in VMware tools, where you can synchronize the VM against the ESX - this we did not use. We let Windows take care of the synchronisation. If you have a mixed environment of DCs (bare metal and virtual), then you can let a bare metal DC sync to an external source, and then let all the other DC's sync to the bare metal DC.


We had the PDC emulator sync with a dedicated physical NTP server, and then let the second DC sync with the PDC emulator. The ESX servers sync with the physical NTP server - but no synchronization between VM and ESX server. Read this article for further info on time sync.

Update: In a KB article (KB 888794) from Microsoft about considerations when hosting DC's in a virtual environment, there is one important paragraph about forced unit access (FUA) which has resulted in some confusion. The paragraph states:

"If the virtual hosting environment software correctly supports a SCSI emulation mode that supports forced unit access (FUA), unbuffered writes that Active Directory performs in this environment are passed to the host operating system. If forced unit access is not supported, you must disable the write cache on all volumes of the guest operating system that host the Active Directory database, the logs, and the checkpoint file. "

According to VMware, forced unit access (FUA) is supported on VMware. Here's the answer from VMware technical support:

-----Original Message-----
From: VMware Technical Support [mailto:webform@vmware.com]
Sent: 24. februar 2010 11:25
To: (Jakob Fabritius Nørregaard)
Subject: Re: VMware Support Request SR# 1490632591

** Please do not change the subject line of this email if you wish to

respond. **

Hello Jakob,

Forced Unit Access is supported by VMware. A large number of customer's have virtualized Domain Controllers which is evident in the community forums.

Thanks & Best Regards

Derek Collins

Technical Support Engineer

VMware Global Support Services

1-877-486-9273

VMware Technical Support Knowledge Base

http://kb.vmware.com/kb"

P2V - Error with NIC after migration with static IP

When doing a P2V and the server has to have a static IP address after the migration, then you may recieve an error message stating that there's an IP conflict and the ip is already configured on an existing NIC - even though only one VMware NIC is visible in network connections.

The reason is that the physical NICs have not been entirely uninstalled in Windows, they still exist in the device manager as hidden devices. Do the following to uninstall the hidden NICs:

Open a command prompt and type the following commands:

set devmgr_show_nonpresent_devices=1
start DEVMGMT.MSC

Click ‘View’ and then click ‘Show Hidden Devices’.
Expand the Network Adapters tree and right click the dimmed network adapter and click ‘Uninstall’ (You may also see a hidden 'RAS async adapter' device under NICs. This cannot be uninstalled. However, it doesn't matter as it doesn't influence the NIC issue, so just leave it).

Now you can configure the static IP with no errors.

Boot a server directly into safe mode

When doing a P2V, it is often necessary to get at VM into safe mode to disable any running HP services and afterwards unistall these HP services (in normal Windows operations). This can either be done by hitting F8 really hard upon boot, or it can be done more smoothly, as hinted by one of my colleagues:

Ones Windows is booted, Start -> Run: msconfig

Tick the /SAFEBOOT option. Reboot the machine, and it will reboot in safe mode. Uncheck this checkmark before returning to regular operations.