How To Analyze PSOD

Written by Suhas Savkoor

Purple Screen of Death or commonly known as PSOD is something which we see most of the times when we run an ESXi host.

Usually when we experience PSOD, we reboot the host (which is a must) and then gather the logs and upload it to VMware support for analysis (where I spend a good amount of time going through it)

Why not take a look at the dumps by yourself?

Step 1:
I am going to simulate a PSOD on my ESXi host. You need to be logged into the host's SSH. The command is

# vsish -e set /reliability/crashMe/Panic 1

And when you open a DCUI to the ESXi host, you can see the PSOD

Step 2:
Sometimes, we might miss out on the screenshot of PSOD. Well that's alright! If we have core-dump configured for the ESXi, we can extract the dump files to gather the crash logs.

Reboot the host, if it is in the PSOD screen. Once the host is back up, login to the SSH/Putty of the host and go to the core directory. The core directory is the location where your PSOD logging go to.

# cd var/core

Then list out the files here:

# ls -lh

Here you can see the vmkernel dump file, and the file is in the zdump format.

Step 3:
How do we extract it?

Well, we have a nice extract script that does all the job, " vmkdump_extract ". This command must be executed against the zdump.1 file, which looks something like this:

# vmkdump_extract vmkernel-zdump.1

It creates four files:
a) vmkernel-log.1
b) vmkernel-core.1
c) visorFS.tar
d) vmkernel-pci

All we require for analysis is the vmkernel-log.1 file

Step 4:
Open the vmkernel-log.1 file using the below command:

# less vmkernel-log.1

Skip to the end of the file by pressing Shift+G. Now let's slowly go to the top by pressing PageUp.
You will come across a line that says @BlueScreen: <event>

In my case, the dumps were:

2015-12-17T20:34:03.603Z cpu3:47209)@BlueScreen: CrashMe 2015-12-17T20:34:03.603Z cpu3:47209)Code start: 0x418021200000 VMK uptime: 0:01:14:16.524> 2015-12-17T20:34:03.603Z cpu3:47209)0x412461a5dc10:[0x41802128d249]PanicvPanicInt@vmkernel#nover+0x575 stack: 0x726f632000000008 2015-12-17T20:34:03.603Z cpu3:47209)0x412461a5dc70:[0x41802128d48d]Panic_NoSave@vmkernel#nover+0x49 stack: 0x412461a5dcd0 2015-12-17T20:34:03.604Z cpu3:47209)0x412461a5dd60:[0x41802157a63b]CrashMeCurrentCore@vmkernel#nover+0x553 stack: 0x100000278 2015-12-17T20:34:03.604Z cpu3:47209)0x412461a5dda0:[0x41802157a8ca]CrashMe_VsiCommandSet@vmkernel#nover+0x13e stack: 0x0 2015-12-17T20:34:03.604Z cpu3:47209)0x412461a5de30:[0x41802160c3c7]VSI_SetInfo@vmkernel#nover+0x2fb stack: 0x41109d630330 2015-12-17T20:34:03.604Z cpu3:47209)0x412461a5dec0:[0x4180217bd7a7]UWVMKSyscallUnpackVSI_Set@<none>#<none>+0xef stack: 0x412461a67000 2015-12-17T20:34:03.604Z cpu3:47209)0x412461a5df00:[0x418021783a47]User_UWVMKSyscallHandler@<none>#<none>+0x243 stack: 0x412461a5df20 2015-12-17T20:34:03.604Z cpu3:47209)0x412461a5df10:[0x4180212aa90d]User_UWVMKSyscallHandler@vmkernel#nover+0x1d stack: 0xffbc0bb8 2015-12-17T20:34:03.604Z cpu3:47209)0x412461a5df20:[0x4180212f2064]gate_entry@vmkernel#nover+0x64 stack: 0x0

• The first line @BlueScreen: Tells the crash exception like Exception 13/14, in my case it is CrashMe which is for a manual crash. 
• The VMKuptime tells the Kernel up-time before the crash.
• The logging after that is the information that we need to be looking for, the cause as to why the crash occurred. 

Now, here the crash dump varies for every crash. These issues can range from hardware errors / driver issues / issues with ESXi build and a lot more.

Each dump analysis would be different. But the basic is the same. 

So, you can try analyzing the dumps by yourself. However, if you are entitled to VMware support, I will do the job for you.

Cheers!

Unable To Take A Snapshot Of A VM - An error occurred while taking a snapshot: Change tracking target file already exists.

Written by Suhas Savkoor

You had a virtual machine which was scheduled for a backup task. The backup job completed and now you want to perform a manual snapshot operation for this virtual machine.

When you take a manual snapshot of this VM, it fails! With the following error;

An error occurred while taking a snapshot: Change tracking target file already exists.

An error occurred while saving the snapshot: Change tracking target file already exists.

There are no snapshots in the snapshot manager. The virtual machine disks are running on the base disk and not a snapshot disk. 

However, when you browse the datastore for this virtual machine, you notice certain ctk.vmdk files. There will be N number of ctk.vmdk for N number of VMDKs. 

A VMware CTK file contains a list of all the changes made to a VMware virtual machine (VM) since it was last backed up. 

CTK change tracking files exist on all VMs where VMware's Changed Block Tracking (CBT) technology is enabled. CBT relies on the CTK file's information to back up only the VM information blocks that have changed, instead of backing up the entire VM. 

There is only one CTK file in the virtual machine disk files, and it has the file extension .ctk. The CTK file is always much smaller than VMware Virtual Machine Disk File (VMDK) block it describes.

What needs to be done?

1. Either browse the datastore for the virtual machine folder from the GUI or open a SSH/Command Line to the host where this virtual machine resides.

2. Delete the ctk.vmdk files

3. Perform the snapshot operation. 

Success!

Configuring VFRC From An Emulated SSD Drive

Written by Suhas Savkoor

If you have read my previous article, it tells you about how to emulate a SSD drive from a Non-SSD disk.

In this article, I will show you the use case of the SSD emulation. I will be configuring VFRC from an emulated SSD drive.

Here, I have created a 2 GB VFRC-Test datastore (VMFS-5) and tagged it as a SSD drive.

Now, if I log in to the web client and select the host and under Manage tab if I select Virtual Flash Resource Management, I see an empty list.
Also, if I go to Add Capacity, I see the list is empty.

Now, even though we have a SSD datastore why is it showing an empty list here.
Well, the requirement for VFRC is a raw SSD disk. A raw disk indicates that there should be no partition on it.
When we added the non SSD disk to VMware vSphere to tag it as a SSD drive, we had to create a VMFS partition on it. This VMFS partition is not allowing the disk to be used by VFRC.

So, what we need to do now is, we will have to remove the partition for this datastore. Please make sure that there is no data on this datastore as it will be lost during the partition removal.

First. we need to determine what is the partition number for our drive.
For this:

Login to the ESXi host via SSH/Putty and run the following command:

# ls /vmfs/devices/disks/

The output is similar to below:

Then, we need to delete the partition. For this, run the below command:

# partedUtil delete "/vmfs/devices/disks/<device_name>" <partition_number>

The output is something as seen below

Here we see we no longer have the partition table listed for this disk. Also, when you come back to your GUI, you no longer see the datastore.

Now, let's login to web client and we can see that this SSD disk is now seen in the VFRC add resource window.

Select the disk and Click OK and you can see the VFRC SSD disk is now added.

Emulate all the way!

Emulating a SSD Drive in VMware 5.x

Written by Suhas Savkoor

You can emulate a SSD disk from a non SSD hard disk.

Just a couple of steps:

Here, I have my Local Datastore, Suhas-Local-4 which is a VMFS-5 datastore and a Non-SSD drive.

Let's tag it.

Step 1:
Take a SSH(Putty) session to the host which has access to the datastore that you want to tag.

Run the command to see if the device is a SSD or not

# esxcli storage core device list --device=<device name>

You can find the name from the GUI; Devices option under Storage
Or you can right click the Datastore > Copy to clipboard and paste it in a notepad to obtain the device name.

The output is similar to this:

Step 2:
We need to add a SATP rule to the device that we want to tag as SSD.

The command is

# esxcli storage nmp satp rule add –satp VMW_SATP_LOCAL –device <device name> --option=enable_ssd

Step 3:
Reboot the host.
Once the host is rebooted from the GUI you can see the datastore is now tagged as SSD

Verify the same from the command line using the command in Step 1

That's it. We have tagged our Non-SSD drive as a SSD disk.
You can now try configuring VFRC (vFlash Read Cache) without an actual SSD.

Upgrading vCenter Appliance 5.x

Written by Suhas Savkoor

Upgrading vCenter Appliance is quite different when compared to upgrading your Windows based vCenter.
There are just a few steps to be performed and a lot to wait for the upgrade to be completed.

Currently, I have a 5.5 Update 3 vCenter and I will be updating it to 5.5 U3a

How to get here?

1. Open a browser and enter:
 
                   https://<Appliance_IP_or_FQDN>:5480

2. Login with root credentials
3. Click Update Tab and Select the Status sub-tab and expand the Details option.


Step 1:

• Login to the host/vCenter hosting this vCenter Appliance virtual machine.
• Right click the Appliance VM and select Open Console

• Select the CD option in tool bar and CD/DVD Drive.
• Since I have my 5.5 Update 3a ISO on the local drive, I will select the option Connect to ISO image on local disk
• Browse to the datastore and select the ISO Image

Step 2:

• Go back to the VCSA management page > Update
• Click the Settings tab: Make sure that the Use CDROM Updates is chosen as we are using the ISO from local drive for upgrade.
• Under Actions > Click Save Settings

Step 3:

• Now the updates or on the CD and the Appliance is set to look for updates on it CDROM folder. 
• Go back to Status Tab and click Check Updates
• A task will run for a couple of seconds and a new Available Updates option is now seen
• If you click the Details here, you can see what update is available. Here, since we mounted the 5.5 U3a update, we can see this in the available updates.
• Click Install Updates

Step 4:

• The update process can take from 90 minutes to 120 minutes. Do NOT reboot or Power OFF the appliance.
• Reboot the machine ONLY after the update is complete. 

During the update process you can see this pop-up in the web management page.

After rebooting the appliance, you can see the new Version Under Appliance Version in Update Tab.
You can also open a console to the Appliance virtual machine and notice the updated Build Number.

Error Logging Into vCenter 6.0 Via vSphere Client With Windows Authentication

Written by Suhas Savkoor

So, you got your new vCenter Appliance 6.0 U1 setup, and you want to dive into managing the environment.
So you fire up a vSphere client, enter the vCenter IP details, and try logging in with Windows Authentication.

And it fails! With this error:

Here, we need to add the vCenter Appliance to a domain.

This is what you have to do:

1. Login to vCenter Appliance console or open a SSH to it. Then change the directory to:

# cd /usr/lib/vmware/likewise/bin/

2. Run this command to join the appliance to a domain:

# ./domainjoin-cli join <domain_name> <domain_username>

3. After executing the above command you will receive a prompt to enter the domain user's password. Enter the password and hit enter.

4. Verify the join by:

# ./domainjoin-cli query

Now you will be able to login to appliance via vSphere Client using Windows Session Credentials.

Configuring VDP 6.x

Written by Suhas Savkoor

Once you have deployed VDP 6.x from the ovf template, you will have to configure the appliance before using it.

To configure VDP you will have to login to the VDP management page. The address would be:

                       https://<vdp_IP_or_FQDN>:8543/vdp-configure

If you receive the error message, "Server has weak ephemeral Diffie-Hellman public key", then follow this article to resolve it.

Step 1:
Once you open the management web page, you will come across this:

Login to the management webpage for VDP configuration.
Username: root
Password: changeme

Once you login, you will encounter VDP configuration wizard.

Step 2:
Click Next and Under network Settings:

1. Enter IPv4 address of VDP appliance
2. Subnet mask and gateway address
3. Primary DNS and an optional secondary DNS.

Add the appliance host-name and Domain. Click Next.

You might receive the following error.

This issue occurs when Fully Qualified Domain Names (FQDN), forward lookup, and reverse lookup are not configured, or they are not configured correctly.

To resolve this, add the VDP appliance name and domain in Forward Lookup in your DNS.

Login to DNS manager > Right click Forward Lookup > Add New Host and provide the Name and IP address of the VDP appliance. Make sure Create associated pointer (PTR) record is checked and select Add Host.

Step 3:
Once successfully added. Select a time zone.

Step 4:
Now, you will receive an option to change your default password "changeme" to a required one.

Step 5:
In the next step, you have to register your VDP appliance to your vCenter machine.

1. Enter vCenter Username. This user must have administrative privileges on the vCenter machine
2. Password for this user
3. vCenter IP and click Test connection.

Once Connection is tested successfully, click Next.

Step 6:
Next is a subjective process where you need to create a new storage for the appliance with the desired size.

Step 7:
Select a datastore where your VDP drives must reside. 3 drives are required, so choose an appropriate datastore with sufficient space.

Step 8:
Enter a required value of CPU and Memory or accept the default ones. Do not lower the default CPU and memory values.

Step 9:
Review the changes and apply the changes. Once the changes are done, the VDP appliance has to be rebooted for the changes to be applied.

And that's it. You got your VDP appliance configured.