To install XenServer 6.0 in a VM, first follow this guide to configure ESXi 5.0 (or watch this youtube video).
One important step is to execute the following command from the console:
echo 'vhv.allow = "TRUE"' >> /etc/vmware/config
Otherwise, configure like the guide. Once the custom VM has been created, to be able to choose ESXi 5 as operating system, go to Edit Settings -> Options -> Guest Operating System choose 'Other' and then choose VMware ESXi 5.x. This will ensure that you won't receive the "HVM is required for this operation" error when trying to boot the win2k8R2 vm (it is possible to change this after the install of XenServer as well).
Add the the XenServer to XenCenter
Create a new VM, choose win2k8 R2 64-bit, mount ISO, install.