Unable To Power On A Virtual Machine: The operation is not allowed in the current connection state of the host

Written by Suhas Savkoor

So I spent a good 30 minutes on a Saturday, on call with AMD as their virtual machine would just not power on. It kept throwing the below error:

Also, you might experience one ore more of the below issues while trying to troubleshoot this virtual machine:

• You are unable to remove this virtual machine from inventory.
• You are unable to vMotion this virtual machine.
• When you login to the host directly via vSphere client and Right click this virtual machine, most of the options are grayed out. Power ON is not available for this virtual machine. 
• The virtual machine does not list when you run 

# vim-cmd vmsvc/getallvms

What can be done to resolve this:

1. Restart the VMware Virtual Center Server service and then try powering ON the virtual machine. 

2. Login to the SSH of the host and restart the management agents using the below command

# services.sh restart

Once the management agents are restarted, the virtual machine might enter into an orphaned state.

3. Right click the virtual machine and remove it from the inventory. This operation will succeed this time.

4. Browse the datastore where this virtual machine is residing and open the virtual machine folder.

5. Right click the virtual machine's .vmx file and add it to the inventory. Select the host/resource pool that is required.

6. Power on the virtual machine and this time the operation should succeed. 

Logging into 5.1 Web Client Displays Associated User's Password Is Expired

Written by Suhas Savkoor

When you open a 5.1 web client, and try logging in, you receive the following error:

The default password expiration time is 365 days. To increase this value:

Step 1:

• Login to SQL Management studio which manages the database of this vCenter. 
• Expand Databases and expand RSA database (Which has information regarding the 5.1 SSO)

Step 2:

• Expand Tables and locate the table: " dbo.IMS_AUTHN_PASSWORD_POLICY "
• Right click this table and select Edit Top 200 Rows 

Step 3:

• Locate the following column: " MAX_LIFE_SEC " and edit the time for password expiration.

          47304000 seconds = 546.5 days
          63072000 seconds = 730 days
          90000000 seconds = 1041days

• Or you can change the " PERIODIC_EXPIRE " column to 0 which implies never expire. 
• Restart the Web Client service and login again, successfully!

Adding Local User and Domain User To ESXi Host

Written by Suhas Savkoor

Every time you login to ESXi host, you login with root credentials. Now, you do not want to use root as a login name, you want to use a user with root permissions to login to the ESXi host. How do we do this?

1. Adding Domain User to ESXi host:
Step 1:
Here you can see that I have logged in as root to my ESXi host.

Step 2:

• Create a Forward Lookup and a Reverse Lookup entry for this ESXi host on your DNS. This is required because, you will have to add the ESXi host to an Active Directory domain. 
• To create a Forward/Reverse lookup record open your DNS manager. 
• Under Forward Lookup Zones, right click your domain and select New Host (A or AAAA)
• Enter the name for the host and the IP of the host. Make sure the "Create associated pointer (PTR) record" is checked.
• Create a Reverse Lookup Zone for this same ESXi host

Step 3:

Verify this information is displayed on the DCUI of the ESXi host. 

Step 4:

• Now we need to add this ESXi to the domain. For this:
• Login to the ESXi host or the vCenter managing this host via a vSphere Client.
• Navigate to Configuration > Authentication Services (Under Software) and click Properties.

• Select Directory Service Type as Active Directory
• Enter the Domain Name and Click Join Domain. It will prompt you for username and the password for the domain. 

Once the Username and Password is Provided, click Join Domain. And Click OK. This will Add the host to the active directory domain. 

Now if you go to Permissions tab and try to add a user from this added domain, you will not be able to see the domain. You will just see (server) domain. To resolve this, you will have to restart the management agents on the ESXi host. Run the below command from the Putty session of the ESXi host. 

# services.sh restart

Once the services are restarted go to Permissions tab, Right click and select Add Permissions. 

Click Add and now from the drop-down you can see the domain. 

• Select a user from the domain and Click Add and click OK. From the Assigned Role Column, select the Administrative Permissions for this user and Click OK. 
• Test the configuration by logging in directly to the host via vSphere client using the AD username and credentials. 
• Test SSH by opening SSH and username will be <domain\username> and Password for the user. 

2. Adding a Local User for ESXi host:

• Login to the host directly via a vSphere Client session.
• Select the host and navigate to "Local Users&Groups tab"
• Right click and Select Add

In the Add New User Window

• Provide a Login name and a Username and a Password for this User. Do not give a UID as this will be automatically created once the user is added. 
• Check Grant shell access for this user option and click OK

Go back to permissions again and add this user and grant him Administrative permissions using the same steps as above. This local user can be found under the (server) domain. 

Test the configuration by opening a Putty Session to the host and login with the user/credentials. 

That's it for today!

Powering OFF a virtual machine fails with the error: Another task is already in progress.

Written by Suhas Savkoor

When you try to Power OFF a virtual machine, you might receive the following error:

Method 1:
Powering OFF the virtual machine from the command line:

From the SSH of the host where this VM resides:
1. Get the VM ID by running the below command:

# vim-cmd vmsvc/getallvms | grep -i <VM_Name>

2. Once the VM ID is obtained, Shutdown the VM guest using the command:

#vim-cmd vmsvc/power.shutdown <VM_ID>

3. If this fails, try a hard shut down of the vvirtual machine using Power OFF command:

#vim-cmd vmsvc/power.off <VM_ID>

4. If this fails as well, perform a vMotion of this VM to another host and perform Method 1 again. If vMotion fails with an error, "This operation is not allowed in the current state", then proceed to step 2, to kill the process

Method 2:
Kill the VM process:

1. Run the command:

ps | grep vmx

The output will be a long list of all the virtual machines vmx process IDs. You need to isolate the part which includes the virtual machine which you want to Power OFF. The output in my environment for a particular VM looks like:

4324194   4324194   vmx                  /bin/vmx
4324200   4324194   vmx-vthread-5:Windows 2008 /bin/vmx
4324201   4324194   vmx-vthread-6:Windows 2008 /bin/vmx
4324202   4324194   vmx-mks:Windows 2008 /bin/vmx
4324203   4324194   vmx-svga:Windows 2008 /bin/vmx
4324204   4324194   vmx-vcpu-0:Windows 2008 /bin/vmx
4324205   4324194   vmx-vcpu-1:Windows 2008 /bin/vmx

The first column contains the Process ID, and second column is Parent Process ID. We need to Kill the parent Process ID. In this example, the Parent Process ID is 4324194.

2. Run the command to kill the process

# kill <process_ID>

3. Also, you can try the below command:

# kill -9 <process_ID>

"-9" is a signal Kill, which sends a sigkill signal to the process that is specified.

Now, all these "kill" command might fail with a message,

"No such process found"Unable to kill Virtual Machine. Error was : Unable to kill virtual machine. Error returned was No such process"

In this case, the only solution to kill this stale process is to Reboot the host having this virtual machine, and this should let us Power OFF the virtual machine.

VMware Labs: ESXtopNGC Plugin Fling

Written by Suhas Savkoor

ESXtop is a command line utility available with vSphere which provides the resource usage of the environment in terms of CPU, memory, disk latency etc. Until now, if you wanted to observe these values, you had to run the command " esxtop " via the SSH of the host.

With this fling, we will now have esxtop integrated as a GUI in the vSphere Web Client.
You can download this fling in this link.

The documentation states that this is available only for Linux based vCenter which is the appliance. However, this can be also applied to your Windows based vCenter.

Procedure:
1. Download the zip file from the provided link.
2. Extract the zip into a desired location.
3. Copy the zip file to the vSphere Web Client installation location. The default location is:

C:\Program Files\VMware\Infrastructure\vSphereWebClient\plugins-packages

4. Restart the Web Client Service.
5. Login to Web Client, select a host, navigate to the Monitor tab and you will notice a sub-tab called " top " which is the esxtop GUI.

You can change the refresh rates of the display counter by modifying the value under the " Set Refresh Rate " option.
You can also export the reading to a csv file by Start/Stop exporting stats option.

Also, you can cycle through CPU, memory, disk, adapter performance tabs and also select which counter values to be displayed by changing the " Select Display Counters " option accordingly.

For a complete list of esxtop counter explanations:
http://www.yellow-bricks.com/esxtop/

The counter update is quite slower when compared to the command line esxtop, but, it is easy to navigate via the GUI.

+1 to this plugin.

VMware Labs: ESXi Embedded Host Client Fling

Written by Suhas Savkoor

Until now, ESXi management was done by two ways. One was by connecting it to a vCenter and logging into this vCenter via a vSphere Client or a Web Client, and the other way, was to directly log into the host via the vSphere Client. This was good, however, I recently came across the ESXi Embedded Host Client, and I found this to be better than the vSphere Client login to host mainly because of two reasons: Easy navigation and no requirement of opening a Putty Session for logging.

You can download the ESXi Embedded Host Client from this link here.

The setup is pretty simple and straightforward. 

1. Enable SSH on the host for which you want to configure the Host Client.

Two paths to proceed further:

Path 1:

In the SSH of the host, run the following command:

 ssh root@<esxip> esxcli software vib install -v <URL>

How to get the URL? 

In the Fling page, on the left side, from the drop-down select the esxui-singed.vib and right click the download button. 

Then go to the SSH session and in the place of <URL>, press Shift+Insert or Right click which will paste the URL. 

Run the command, and it will prompt for the root password. The VIB installation completion status will be displayed. 

Path 2:

Offline Install:

If you are using a 5.x host, from the drop down choose the 5.x bundle, otherwise the 6.x bundle. 

Check Agree and download box and click download. 

Login to the host via the vSphere client and upload the VIB to a datastore.

Open a SSH to this host and run the below command:

scp /path/to/downloaded/vib/esxui.vib root@<esxip>:/tmp

This looks something like below for my environment:

scp /vmfs/volumes/Protected_LUN/VMware_bootbank_esx-ui_0.0.2-0.1.3357452.vib root@192.168.1.176:/tmp 

Enter the Password when prompted. This will import the VIB from the datastore to the tmp folder of the host. 

Then to install the VIB run the following command:

ssh root@<esxip> esxcli software vib install -v /tmp/esxui.vib 

It will ask for the password of the ESXi host. Once completed the output is something like this:

Host reboot is not required.

To manage this:

Open a browser and enter the URL: https://<host_IP>/ui
You might receive: 503 Service unavailable

To resolve this:
1. Go back to the SSH session of this host.
2. Go to the following directory

# cd /etc/vmware/rhttpproxy/

3. Here you will see a file called endpoints.conf
Open this file using an editor

#vi endpoints.conf

4. Press "i" to begin edit and remove the following line:
/ui local 8308 redirect allow
5. Save the file by pressing ESC and then typing :wq!
6. Restart the rhttpproxy service using the below command:

/etc/init.d/rhttpporxy restart 

Login to the client again using the same address and you will be presented with the login screen.

Logging in with root credentials will give you the management page:

Now, there are lot of options here that you can explore from deploying VMs to managing the host.
The best part I liked about this is the ESXi logs. If you click the Monitor option on the left side and go to Logs tab you will get the list of ESXi logs that are available. Upon selecting the logs you can view that particular logging in the window below.

The host client is great and I will be actively testing this and will update this article if needed be.

Cheers!

vCenter 6.0 deployment models

Written by Suhas Savkoor

So with 5.1 and 5.5, the vCenter components included were Single Sign On, Web Client, Inventory Service and the vCenter Server. These components could be deployed all on one single machine, as a simple install. If we wanted redundancy, or removing one single point of failure, we had the choice to deploy each component on a different machine. We also had a choice to mix the installation, grouping certain components on certain set of machines. 

With vCenter 6.0, we have a concept of Platform Service Controller (PSC) and Management Server Node. With PSC and Management server into picture, most of the vCenter 5.x components have been consolidated into these two services.

The PSC includes the below services:

• VMware Single Sign On
• VMware License Service
• VMware Lookup Service and Directory Service
• VMware Certificate Authority

The Management Server node includes the following:

• vCenter Server
• Web Client 
• Inventory Service
• Other Services that were seen in the 5.5 installation ISO: (Auto Deploy, Dump Collector, Syslog Collector)

With these two components in picture, the vCenter 6.0 has a flexible deployment topology. Let's take a look at couple of recommended and non recommended types:

Recommended Types:

1. vCenter Server with Embedded PSC:

In embedded PSC deployment, you have the PSC component and the vCenter component residing on the same machine. This can be a physical machine or a virtual machine managed by an ESXi host. 

This type of deployment is used for small environments and it is the easiest of all deployments. Maintaining embedded PSC deployment is pretty easy. However, it comes with certain drawbacks such as single point of failure and replication between multiple embedded nodes is not supported. 

This type of deployment is supported both on Windows and Appliance based vCenter.

2. vCenter Server with External PSC:

In this type of deployment, the PSC is deployed on a separate machine (physical/virtual)

Using this type of deployment, you can have multiple vCenter Servers (Management server node) being managed by or connected to one PSC node. 

Using this mode, you can deploy a vCenter Server in a linked mode with two or more vCenter Server connected to the same PSC instance. Both the vCenter Servers are connected to the same PSC domain. 

This type of deployment is supported both on Windows and Appliance based vCenter.

This type of deployment is also known as Enhanced Linked Mode with External PSC Without HA. 

3. vCenter Server with External PSC in HA:

In the previous deployment type, if the PSC node goes down, then the vCenter authentication fails. In PSC with HA type deployment, the PSC node is made fault-tolerant by adding another PSC node. Both these PSC nodes reside on a VM of their own and is configured to reside behind a Load Balancer. The vCenter Servers are joined to the PSC domain using the load balancer IP which is shared among the multiple PSC nodes. So, in this case, even if one PSC node goes down, the authentication for vCenter nodes can be provided from the other PSC nodes. 

This configuration can be configured again for both Windows and Appliance base vCenter.

These 3 types of deployments are supported and recommended by VMware. 

There are couple of more types of vCenter deployment models which are supported, however, not recommended by VMware. Let's take a look at the these deployments:

Non Recommended Types:

1. Enhanced Linked Mode with Embedded PSC:

As we saw earlier embedded PSC with Linked Mode is not recommended. However, the deployment can still be done for a test environment. Here you deploy your first embedded node (PSC and Management Node) on a single physical/virtual machine. You then deploy the other embedded nodes, with their embedded PSC configured to the domain of the first embedded Node. 

2. Combination of Embedded PSC and External PSC:

In this type of deployment, the first node is an embedded node. On the first node, the PSC and Management Server are deployed on the same virtual machine. 

The second node is an External PSC node, in which the PSC is deployed on a separate machine, and is joined to the domain of the Embedded PSC. Multiple management nodes can be deployed and connected to the secondary PSC (External)

3. Enhanced linked mode using Embedded PSC only:

In this type of deployment, the first node is again an embedded PSC node. The second node does not have a PSC. The second node is just a Management Server node, and this node is connected to the embedded PSC domain. 

Bottom Line:

If there is just one vCenter instance that is required then you can go for embedded PSC deployment. You can make node Fault-Tolerant by using VMware HA or Fault Tolerance. Small environments will fall into the category of embedded deployment models. 

Upgrade Basics from 5.1 or 5.5 to 6.0 

1. Upgrade Simple Install 5.1/5.5 to Embedded PSC 6.0

In this, currently you have a simple install of a 5.1/5.5 vCenter. That is, SSO, Web Client, Inventory Service and vCenter all on the same machine. This vCenter machine needs to be upgraded to 6.0 with Embedded PSC. As mentioned above, in embedded PSC the vCenter and SSO all reside on the same machine.

So the upgrade process would be straightforward. Mount the ISO and upgrade the PSC first and then the management next. 

2. Upgrading Simple Install 5.1/5.5 to External PSC 6.0

In this scenario the 5.1/5.5 again is a simple install. However, when upgrading to 6.0, you want the PSC to be on a separate machine to provide Enhanced Linked Mode capabilities.

In this case:

• Deploy a new Windows Server machine and Install the current version of SSO only on this machine
• Re-point your Web Client, Inventory Service and vCenter Server to this new instance of SSO that was installed on the separate machine
• Upgrade the SSO node first to 6.0
• Upgrade the vCenter node next to 6.0
• Uninstall the older version of 5.x SSO that was residing on the original simple install machine as this is not being used anymore. 

More information and detailed steps for Install and Upgrades can be found in the below link:

https://featurewalkthrough.vmware.com/#!/vsphere-6-0

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2109772

Picture credits to PenguinPunk