Finding Lock On A File

Written by Suhas Savkoor

If we want to determine which ESXi host is holding a lock on the file, we usually end up following this KB article. The classic command we use is vmkfstools -D /path/to/the/file

I came across another command, which is pretty good and also saves the time for us in finding out which host corresponds to the MAC address that is locking the particular file. From the SSH of the host, browse the virtual machine directory. 

Once you are inside the virtual machine's directory run the following command against the file, that you want to check for lock. 

vmfsfilelockinfo -p <file_name> 

For example:

# cd /vmfs/volumes/52a62585-b8e592b8-3570-005056030027/CentOS7 

# vmfsfilelockinfo -p CentOS7-flat.vmdk 

The output is as follows:

vmfsfilelockinfo Version 1.0

Looking for lock owners on "CentOS7-flat.vmdk"

"CentOS7-flat.vmdk" is locked in Exclusive mode by host having mac address ['00:50:56:03:0d:38']

Trying to make use of Fault Domain Manager

----------------------------------------------------------------------

Found 3 ESX hosts using Fault Domain Manager.

----------------------------------------------------------------------

Searching on Host 192.168.1.177

Searching on Host 192.168.1.176

    MAC Address : 00:50:56:03:0d:38

Host owning the lock on the vmdk is 192.168.1.176, lockMode : Exclusive

Total time taken : 0.02 seconds.

This tells us the MAC address of the host having the lock on the file, as well as which host corresponds to that MAC address even if the SSH session is conducted through a different ESXi host.

This will again not work for NFS!

vSphere Install and Upgrade Series

Written by Suhas Savkoor

Welcome to VMware vSphere Install and Upgrade Series. This section will discuss about fresh installation as well as upgrading VMware vCenter from 5.1 to 5.5 to 6.0 (Embedded and External PSC) on both appliance as well as Windows vCenter level. It also discusses about fresh installations of 5.x and 6.x version of vCenter.

**Note** This article will be updated with new links as and when the setup is completed.

Fresh Install Series:

1. Fresh installation: Embedded PSC deployment of a 6.0 vCenter Server Appliance
2. Fresh Installation: Simple Install of a 5.5 vCenter Server (Windows)
3. Fresh Installation: External PSC deployment of a 6.0 vCenter Server Appliance

Upgrade Series

1. Upgrade: vCenter Server Appliance from 6.0 to 6.0 U1 with embedded PSC
2. Upgrade: vCenter Server Appliance from 6.0 to 6.0 U1 with external PSC

Installing vCenter Appliance 6.0 with Embedded PSC

Written by Suhas Savkoor

vCenter appliance as you know is a SUSE Linux alternative to Windows based vCenter. This article talks about Embedded PSC deployment of a new 6.0 appliance. The embedded PSC has both the PSC node and the management vCenter node on the same machine.

More about vCenter deployment models can be found in this link

Step 1:
Download the vCenter 6.0 appliance ISO . Click here for the download link
Second option in the download section is the appliance ISO

Pre-requirements:

• You will need a Windows machine which acts as a mount point for the vCenter appliance ISO. Using this mount point you can deploy the appliance on to an ESXi host, or a vCenter (if you already have one)
• The forward lookup and reverse lookup for the appliance name should be added in your DNS manager. If this is not done, the installation fails in mid progress. 
• So, plan out a name for your appliance and an IP address for the same. Go to your DNS manager and enter this name and the IP address for your forward lookup zone and the network ID for the reverse lookup zone. 
• Make sure that there is sufficient space on the datastore.

Step 2:

• Mount the ISO on the Windows virtual machine. 
• Open the Drive and you will see the following contents. Here, the Client Integration Plugin has to be installed first as the web browser is going to be used for the installation 
• Open the VCSA folder. 

Inside this you will have the Client Integration Plugin setup .exe file. Run this file. 

You will come across the installation wizard of the Plugin. This is a simple installation; accept the EULA and proceed to install. 

Step 3:

Once the plugin installation is done, go back to the root directory of the vcsa disk drive and you will find vcsa-setup.html file. 

Open this file to continue further with installation.

You will now come across the install/upgrade web page. 

Click Install to begin the setup and deployment of the 6.0 appliance.

Step 4:

Accept the End User License Agreement and click Next

Step 5:

Here, we need to provide the details of the ESXi host on which you want to deploy your new appliance. 

• Provide ESXi host's IP address or Fully Qualified Domain Name
• Username for the ESXi, root
• And the respective password

If it asks for a Security Thumbprint, click Yes to proceed.

Step 6:

Here you will provide your appliance details. 

• For appliance name: Enter any name for this appliance. However, make sure that this name is added to the Forward Lookup Zone in your DNS manager. 
• Provide a password for the " root " user of the appliance. 

*Note* Document this password as this would be required for upgrading the appliance in future. 

Step 7:

Since here we are doing an Embedded PSC install. Choose the first (default) option.

Step 8:

Since this is a fresh install, we will be creating a new SSO domain. The details to be filled are:

• Password: Enter a new password for the SSO user. This user is used to manage your Identity sources, users and groups configuration for your vCenter appliance. Please document this password as well, as it is necessary.
• SSO domain name: This has to be a unique domain name. It can be anything, here I am using the "vsphere.local" as my SSO domain. The SSO user then would be administrator@vsphere.local

*Note* If you have an existing domain in your active directory, say, xyz.com, DO NOT use this domain as your SSO domain. This will prevent you from adding users under to xyz.com domain to the vCenter for management. 

• The SSO Site Name: This can be any user defined site name. 

Step 9:

Choose the appliance size depending on the number of hosts and VMs you have in your inventory.

Step 10:

Select a datastore where the appliance files must reside. 

Step 11:

Choose a database type:

• If it is embedded, a vPostgres DB is installed
• If you have an external Oracle DB, this can also be specified. 

Step 12:

Here you will enter the network settings for the appliance:

• Select the port-group where this appliance must reside on the ESXi networking
• Enter an IP address to this appliance. This IP address should correspond to the appliance name that was added in the Forward Lookup Zone of the DNS manager
• The FQDN of the appliance, the appliance name and your domain name
• Subnet mask, gateway and DNS server details.
• If you have a separate server for your NTP, then check "Use NTP Servers" and paste the IP in the below box.

Click Next, review the settings and Finish to begin the installation/deployment. 

Once the installation is complete, you will see the following screen with a link to Web Client. Click the link and login with SSO credentials to verify everything is working good.

That's pretty much about installation of an embedded PSC appliance 6.0

Unable To Power On A Virtual Machine: The operation is not allowed in the current connection state of the host

Written by Suhas Savkoor

So I spent a good 30 minutes on a Saturday, on call with AMD as their virtual machine would just not power on. It kept throwing the below error:

Also, you might experience one ore more of the below issues while trying to troubleshoot this virtual machine:

• You are unable to remove this virtual machine from inventory.
• You are unable to vMotion this virtual machine.
• When you login to the host directly via vSphere client and Right click this virtual machine, most of the options are grayed out. Power ON is not available for this virtual machine. 
• The virtual machine does not list when you run 

# vim-cmd vmsvc/getallvms

What can be done to resolve this:

1. Restart the VMware Virtual Center Server service and then try powering ON the virtual machine. 

2. Login to the SSH of the host and restart the management agents using the below command

# services.sh restart

Once the management agents are restarted, the virtual machine might enter into an orphaned state.

3. Right click the virtual machine and remove it from the inventory. This operation will succeed this time.

4. Browse the datastore where this virtual machine is residing and open the virtual machine folder.

5. Right click the virtual machine's .vmx file and add it to the inventory. Select the host/resource pool that is required.

6. Power on the virtual machine and this time the operation should succeed. 

Logging into 5.1 Web Client Displays Associated User's Password Is Expired

Written by Suhas Savkoor

When you open a 5.1 web client, and try logging in, you receive the following error:

The default password expiration time is 365 days. To increase this value:

Step 1:

• Login to SQL Management studio which manages the database of this vCenter. 
• Expand Databases and expand RSA database (Which has information regarding the 5.1 SSO)

Step 2:

• Expand Tables and locate the table: " dbo.IMS_AUTHN_PASSWORD_POLICY "
• Right click this table and select Edit Top 200 Rows 

Step 3:

• Locate the following column: " MAX_LIFE_SEC " and edit the time for password expiration.

          47304000 seconds = 546.5 days
          63072000 seconds = 730 days
          90000000 seconds = 1041days

• Or you can change the " PERIODIC_EXPIRE " column to 0 which implies never expire. 
• Restart the Web Client service and login again, successfully!

Adding Local User and Domain User To ESXi Host

Written by Suhas Savkoor

Every time you login to ESXi host, you login with root credentials. Now, you do not want to use root as a login name, you want to use a user with root permissions to login to the ESXi host. How do we do this?

1. Adding Domain User to ESXi host:
Step 1:
Here you can see that I have logged in as root to my ESXi host.

Step 2:

• Create a Forward Lookup and a Reverse Lookup entry for this ESXi host on your DNS. This is required because, you will have to add the ESXi host to an Active Directory domain. 
• To create a Forward/Reverse lookup record open your DNS manager. 
• Under Forward Lookup Zones, right click your domain and select New Host (A or AAAA)
• Enter the name for the host and the IP of the host. Make sure the "Create associated pointer (PTR) record" is checked.
• Create a Reverse Lookup Zone for this same ESXi host

Step 3:

Verify this information is displayed on the DCUI of the ESXi host. 

Step 4:

• Now we need to add this ESXi to the domain. For this:
• Login to the ESXi host or the vCenter managing this host via a vSphere Client.
• Navigate to Configuration > Authentication Services (Under Software) and click Properties.

• Select Directory Service Type as Active Directory
• Enter the Domain Name and Click Join Domain. It will prompt you for username and the password for the domain. 

Once the Username and Password is Provided, click Join Domain. And Click OK. This will Add the host to the active directory domain. 

Now if you go to Permissions tab and try to add a user from this added domain, you will not be able to see the domain. You will just see (server) domain. To resolve this, you will have to restart the management agents on the ESXi host. Run the below command from the Putty session of the ESXi host. 

# services.sh restart

Once the services are restarted go to Permissions tab, Right click and select Add Permissions. 

Click Add and now from the drop-down you can see the domain. 

• Select a user from the domain and Click Add and click OK. From the Assigned Role Column, select the Administrative Permissions for this user and Click OK. 
• Test the configuration by logging in directly to the host via vSphere client using the AD username and credentials. 
• Test SSH by opening SSH and username will be <domain\username> and Password for the user. 

2. Adding a Local User for ESXi host:

• Login to the host directly via a vSphere Client session.
• Select the host and navigate to "Local Users&Groups tab"
• Right click and Select Add

In the Add New User Window

• Provide a Login name and a Username and a Password for this User. Do not give a UID as this will be automatically created once the user is added. 
• Check Grant shell access for this user option and click OK

Go back to permissions again and add this user and grant him Administrative permissions using the same steps as above. This local user can be found under the (server) domain. 

Test the configuration by opening a Putty Session to the host and login with the user/credentials. 

That's it for today!

Powering OFF a virtual machine fails with the error: Another task is already in progress.

Written by Suhas Savkoor

When you try to Power OFF a virtual machine, you might receive the following error:

Method 1:
Powering OFF the virtual machine from the command line:

From the SSH of the host where this VM resides:
1. Get the VM ID by running the below command:

# vim-cmd vmsvc/getallvms | grep -i <VM_Name>

2. Once the VM ID is obtained, Shutdown the VM guest using the command:

#vim-cmd vmsvc/power.shutdown <VM_ID>

3. If this fails, try a hard shut down of the vvirtual machine using Power OFF command:

#vim-cmd vmsvc/power.off <VM_ID>

4. If this fails as well, perform a vMotion of this VM to another host and perform Method 1 again. If vMotion fails with an error, "This operation is not allowed in the current state", then proceed to step 2, to kill the process

Method 2:
Kill the VM process:

1. Run the command:

ps | grep vmx

The output will be a long list of all the virtual machines vmx process IDs. You need to isolate the part which includes the virtual machine which you want to Power OFF. The output in my environment for a particular VM looks like:

4324194   4324194   vmx                  /bin/vmx
4324200   4324194   vmx-vthread-5:Windows 2008 /bin/vmx
4324201   4324194   vmx-vthread-6:Windows 2008 /bin/vmx
4324202   4324194   vmx-mks:Windows 2008 /bin/vmx
4324203   4324194   vmx-svga:Windows 2008 /bin/vmx
4324204   4324194   vmx-vcpu-0:Windows 2008 /bin/vmx
4324205   4324194   vmx-vcpu-1:Windows 2008 /bin/vmx

The first column contains the Process ID, and second column is Parent Process ID. We need to Kill the parent Process ID. In this example, the Parent Process ID is 4324194.

2. Run the command to kill the process

# kill <process_ID>

3. Also, you can try the below command:

# kill -9 <process_ID>

"-9" is a signal Kill, which sends a sigkill signal to the process that is specified.

Now, all these "kill" command might fail with a message,

"No such process found"Unable to kill Virtual Machine. Error was : Unable to kill virtual machine. Error returned was No such process"

In this case, the only solution to kill this stale process is to Reboot the host having this virtual machine, and this should let us Power OFF the virtual machine.