Howto: Permission wars in VI3

UPDATE: This setup doesn't entirely work. Templates aren't visible to the users...

This past week, I have been working on an interesting problem. A new internal customer wanted a development environment where they could free hands to deploy and delete VMs, take snapshots etc. To more or less have free hands and the VMware team should provide the virtual infrastructure as a service.

Now, from a virtual infrastructure operations perspective, to give a customer that much freedom is a bit of an administrative nightmare. For example, how do you ensure that a cluster is not overcommitted and how to make sure that all servers are properly registered in the CMDB.

To address the most important issue - from a technical perspective: The customer should not be able to overcommit the cluster. If they have that possibility, then we can't do maintenance, there won't be full failover. The obvious way to go about it is to enable HA and then to check the 'Prevent VMs from being powered on if they violate availability constraints'. However, HA does not have the most sensible way of calculating HA slot sizes and if you only have two hosts in a cluster, then you risk not being able to deploy a new VM even though there are plenty of resources in the cluster.

A colleague of mine suggested that I create a root resource pool in the cluster and then add permissions only on that resource pool and not on the host, cluster, or datacenter level. In theory, this is a pretty good idea, as you can set a hard limit on the resource pool for memory usage (which in my experience is the typical, visible, limiting factor in the cluster). In this case, I set a limit of 50% of available memory and then made the resource pool non-expandable. The resource pool limits in relation to actually used memory - not what is assigned to the VMs, see below.

I created a role similar (I think ;-)) to virtual machine administrator, which can more or less anything at the virtual machine layer (deploy, delete, change, snapshot, mount ISO's etc.) and added this at the resource pool layer. When I started testing, I discovered a number of issues. First, I couldn't create a VM, I couldn't delete a VM, and I couldn't browse the datastore from the VM summary page. But these permissions were already given to the role. If the same role was applied to the cluster or datacenter level, then it worked fine. So it makes a difference at which level the permisssions are applied.

If I apply the role at the cluster level, then everything works in an acces rights perspective, but then the role have too many permissions. Then, they can deploy servers directly in the cluster and will not be forced to deploy into the root resource pool. And then control is lost.

The only way I could work around this issue was to create two seperate role with two different permission sets and then apply them at two different levels of the datacenter.

The first role has very few permissions and is applied at the datacenter level (do not propagate) (this could also be at cluster level, but currently I only have one cluster in the datacenter...). The second role is the actual role that I created in the first place. This role was applied to the cluster level (propagate rights) where a hard limit has been defined for memory.

Below is listed the permission mapping that I have used for both roles.

With setup, the user is completely locked down, so they can only deploy servers in the defined resource pool and they will not be allowed to overcommit. If they do, the VM's won't be able to power on.

In relation to snapshots and running out of space on the LUN, this problems still persists but will not be addressed in this article.

Role 1 (do not propagate rights) - to be applied at datacenter level

Virtual Machine.Inventory.Create

Virtual Machine.Inventory.Remove (otherwise one can’t delete VM from disk)

Virtual Machine.Configuration.Add New Disk

Datastore.Browse Datastore (to be able to browse datastore from VM summary view)

Role 2 (propagate rights) - to be applied at resource pool level

Datastore.Browse Datastore

Datastore.File Management

Virtual Machine.Inventory.Create

Virtual Machine.Inventory.Remove

Virtual Machine.Inventory.Move

Virtual Machine.Interaction.Power On

Virtual Machine.Interaction.Power Off

Virtual Machine.Interaction.Reset

Virtual Machine.Interaction.Answer Question

Virtual Machine.Interaction.Console Interaction

Virtual Machine.Interaction.Device Connection

Virtual Machine.Interaction.Configure CD Media

Virtual Machine.Interaction.Tools Install

Virtual Machine.Configuration.Rename

Virtual Machine.Configuration.Add Existing Disk

Virtual Machine.Configuration.Add New Disk

Virtual Machine.Configuration.Remove Disk

Virtual Machine.Configuration.Change CPU Count

Virtual Machine.Configuration.Memory

Virtual Machine.Configuration.Add or Remove Device

Virtual Machine.Configuration.Modify Device Settings

Virtual Machine.Configuration.Settings

Virtual Machine.Configuration.Change Resource

Virtual Machine.Configuration.Reset Guest Information

Virtual Machine.Configuration.DiskExtend

Virtual Machine.State.Create Snapshot

Virtual Machine.State.Revert to Snapshot

Virtual Machine.State.Remove Snapshot

Virtual Machine.State.Rename Snapshot

Virtual Machine.Provisioning.Customize

Virtual Machine.Provisioning.Clone

Virtual Machine.Provisioning.Create Template From Virtual Machine

Virtual Machine.Provisioning.Deploy Template

Virtual Machine.Provisioning.Clone Template

Virtual Machine.Provisioning.Mark as Template

Virtual Machine.Provisioning.Mark as Virtual Machine

Virtual Machine.Provisioning.Read Customization Specifications

Virtual Machine.Provisioning.Allow Virtual Machine Download

Virtual Machine.Provisioning.Allow Virtual Machine Files Upload

Resource.Assign Virtual Machine to Resource Pool

Resource.Migrate

Resource.Relocate

Supported image formats for Converter 4 Standalone

VMware Converter 4 Standalone supports the following image formats (see screen dump below). Hyper-V is supported but conversion will have to be done as a physical server (see link)

http://www.vmware.com/products/converter/faqs.html

Problems showing performance stats in VC after DB upgrade

We had an incident the other day about a VirtualCenter v2.5 U4 not showing performance statistics for the VM's. It was possible to see live stats for e.g. CPU and memory usage. But changing the chart options to 'weekly' or 'monthly' resulted in a 'Performance data is currently not available for this entity'.

Recently the backend SQL Express server for this VC had been upgraded to a SQL 2005 Standard edition and this was the reason for the error.

In the SQL server, there are three stat-roolup-jobs (which are the ones creating the perf stats in VC) which were not automatically created during the upgrade. These had to be added manually following this KB article from VMware. These are:

• Past Day stats rollup
• Past Week stats rollup
• Past Month stats rollup

After adding the jobs and waiting a couple of hours for all of the jobs to have run, everything worked just fine (The VC DB is called VCDB - UMDB is for Update Manager).

Below is a screendump from the SQL Management Studio after the jobs had been added:

Howto: Removing a disk from a VM - howto identy the right disk?

From time to time, we need to remove disks from a VM. If there's only two or three disks attached to the VM, it's typically not a problem figuring out which one to remove e.g. if the disks have different sizes. But if you have seven or eight disks and they are the same size, then it's a bit more tricky - let's say if you're asked to remove the 'E-drive'. Under 'Edit Settings' for the VM, the disks only have a number which does not necessarily correspond with anything within the VM.

So how to identify exactly which disk that corresponds with a given volume within Windows?

The match can be made by looking at the SCSI target ID for the disk - this can be identified both in WIndows and under 'Edit settings' for the VM (A VM can have four SCSI controllers with up to 15 disks on each controller, so a maximum of 60 disks per VM).

To identify SCSI target ID within the VM:

Go to Computer Management -> Disk Management

Right click a disk and choose Properties

On the General tab you will see the Bus number (SCSI controller) and the Target ID (SCSI target ID), note the number - in this case below the ID is 4.

To identify SCSI target ID from the VI client:

Now go to 'Edit Settings' for the VM under and locate the disk with the corresponding target ID (see Virtual Device Node for the disk). Make sure the that the controller number and SCSI ID is the same. In this case it is Hard Disk 5 that have SCSI ID 4.

Shut down the VM to remove the disk.

SVMotion GUI plugin for VI Client in VI3

Lost Creations has made this very popular GUI plugin for doing SVMotion from the VI Client. It has been out there for quite some time, so this post is merely for my own reference (I actually thought I had posted about this before...)

It's absolutely a 'must have' tool for daily operations of the virtual infrastructure.

http://lostcreations.com/viplugins

(Update 2011.01.05: Use this link for download in stead)

Go here for installation guide.

SVMotion with .vmdk's on different LUN's

Yesterday, I had to extend a number of disks on a VM. There were about seven .vmdk's spread over three different LUN's which were all out of space. In VI3 there's really no good way to increase a LUN (unless you use extend, but don't), so to increase the disk sizes of the .vmdk's, a larger LUN had to be created onto which the .vmdk's could be moved before extending them.

The storage guys create a 1 TB LUN for the VM. So, I wanted to use SVMotion to move the .vmdk's one by one to the new LUN. If you start out with a disk that is not the primary, or OS, disk you will get an error (I'm using the GUI plugin from Lost Creations), so you can only move the primary disk. However, when you move that primary disk, all of the .vmdk's attached to that VM will be moved with the VM at the same time and will be placed on the target LUN.

So when SVMotioning, all .vmdk's attached to that VM are moved at the same time. Therefore, make sure to have enough space on the target LUN.