Unable To Add A Client To A Backup Job In VDP: Already Registered

So today I was working on a case where a vCenter virtual machine and a PSC virtual machine were added to a backup job. One fine day, the PSC machine started behaving weirdly and had to be discarded. So the end solution was to re-deploy the PSC machine completely. The name of the faulty PSC machine was XYZ. This was renamed to XYZ-Old and a new PSC was deployed with the same old name, XYZ. 

The vCenter was back up and running good now. So, we went to the web client and connected to the VDP plugin and discarded the old backup job. So now, the backup tab had no backup jobs at all. Then, we went ahead and created a new backup job and added the new PSC machine, XYZ to this and clicked Finish. The screen grayed out for few seconds and the task finished with an error stating:

Backup job was created successfully, but the system failed to add the following clients to the job. ERROR: Host Named (<host-name>) is already registered as........

Why this is caused:

It is saying that the PSC, XYZ is already registered because it still had the old machine entry in the protected client list in the VDP backup client domain. This stale entry was preventing us to add the new machine with the old name, even though the old PSC machine was renamed and the backup job corresponding to it was removed. 

The solution:

1. Open a SSH to the VDP appliance and run the below command:

mccli client show --recursive

This will show all the protected clients associated with this VDP appliance.

Here I noticed that the XYZ PSC machine entry was still present. I need to remove this stale entry.

2. Run the below command to retire that VM from the protected list 

mccli client retire --domain=/vCenter-IP/VirtualMachines --name="VM-Name"

You will receive an output saying virtual machine is retired successfully. 

Once this was done, we went ahead and added the XYZ PSC virtual machine to the backup job successfully. 

Sharing is caring! You would not find much of VDP articles, believe me!!

Thank you!

Unable to vMotion Due To Difference In vDS Vendor Information Between Source And Destination vDS

So I was trying to vMotion my virtual machines between two vDS and it was failing with the following error: 

"The destination distributed switch has a different version or vendor than the source distributed switch"

A quick search brings up the following KB article which is great, however, there are certain cases where people do not want to upgrade the DVS to change the vendor ID. 

In that situation, there are two workarounds:

1. Changing the entry from the MOB page of the vCenter:

• Open a browser and go to the following IP address to open the Managed Object Browser page:

https://<vcenter-IP/mob

• Click Content
• Under Name: rootFolder select the value: group-d1 (Datacenters)
• Under Name: childEntity select the value data-center
• Under Name: networkFolder select group-n* (network)
• Under Name: childEntity locate the DV Switch and prior to the name of the DVSwitch you will have the moid of the Switch in the format dvs-*number*
• Go to the following location once the moID is obtained. 

                                         https://vCenter-IP/mob/?moid=dvs-141

• Click PerformDvsProductSpecOperation_Task on the page
• Enter the following, For Fill operation: Upgrade
• Fill productSpec:

                <productSpec>

                     <vendor>VMware, Inc.</vendor>

               </productSpec>

• Click Invoke Method

Go back to the vCenter Server and verify the Vendor Name now on your vDS. 

2. If you are using a SQL database, you can run the below query: 

update vpx_dvs set product_vendor='VMware, Inc.' where id=XXX(replace XXX with your dvs id);

dvs ID can be found from the steps above. 

However, before performing any steps, have snapshots, backup of your vCenter and database. 

VDP Full and File Level Restore Fails: Failed to get disks: Unable to browse as proxies are unavailable

So when trying to perform a full virtual machine restore or a File Level Restore for any virtual machine the task fails, at various possible percentages. The failure descried here is, 

Failed to get disks: Unable to browse as proxies are unavailable

If I try to restore this virtual machine on any other host, datastore or restore as a new virtual machine, replace existing virtual machine, the task still fails with the same error. 

The restore logs, in the following location shows:

Location: # cd /usr/local/avamarclient/var

The log with following MOD-*EPOCH*-vmimage1.log would be the restore logs:

The logs had the following;

2016-05-23T16:01:39.723-02:00 avvcbimage Error <0000>: [IMG0011] Timeout on wait for spawned restore metadata avtar process to complete

2016-05-23T16:06:11.947-02:00 avvcbimage FATAL <17824>: GetDiskAttributed Failed

2016-05-23T16:06:12.121-02:00 avvcbimage Error <17771>: Invalid request to create a VM.

2016-05-23T16:06:12.121-02:00 avvcbimage Error <0000>: [IMG2012] VM creation failed during restore.

Here the timeout is occurring while the AvVcbimage is spawning the avTar for metadata restore. This in-turn is failing to create the virtual machine for restore. 

The resolution:

Increase the AvVcbimage timeout using the below steps:

1. If the VDP is using external proxy, then you will have to SSH to your proxy machine. If the VDP is using internal proxy, then we will have to use the SSH to VDP appliance. Change the directory to:

# cd /usr/local/avamarclient/var

2. Edit the following file:

# vi vvcbimage.cmd

3. Add the below and save the file

--subprocesstimeout=600

4. Restart the avAgent service using the below command:

# avagent-vmware restart

Run the restore operation now, and it should work good. 

If not, leave a comment! Thank you.

Fatal Error While Creating Storage For A New VDP Appliance Deployment

After you deploy the ova template for VDP we need to configure the VDP appliance from the vdp-configure page. During the storage configuration, I had chosen about 8 TB of storage for the VDP and during the configuration progress, the setup encountered a Fatal Error. The exact message and the screenshot is:

“A fatal error occurred during the storage configuration and the appliance is unrecoverable. This could be due to the datastore not having enough free space, does not support the required VMDK size, or is in inactive state. Please check the vCenter Tasks pane for the exact error and redeploy a new VDP appliance”

Couple of pre-checks:

1. Make sure the destination VMFS / NFS datastore has sufficient space to accommodate the selected VDP data storage disks. 

2. If we are using VMFS datastore, make sure that the block size for the VMFS datastore supports the VMDK size being created on it

3. Then lastly, check for the permissions for the user that the VDP is being registered to vCenter with, also called as VDP user. The user should have disk create role, or best yet, administrator role. 

So, I used a domain user with administrator privilege or you can also use the SSO user to get the VDP appliance to registered to the vCenter server. 

The disk creation completed successfully and I was able to continue to use the VDP appliance. 

Unable To Connect VDP 6.1 To Web Client

So once the new VDP 6.1 appliance is deployed, we login to Web Client and select the vSphere Data Protection plugin. From the drop-down, we will select the required VDP appliance and click connect. However, upon performing this, the screen grays out forever, and this operation does not fail either with an error. 

The thing to notice here is:

1. This issue occurs when the VDP 6.1 is on a distributed switch. 

2. If the VDP virtual machine is migrated to a standard switch, the appliance is able to connect to the web client

3. If the VMs that need to be backed up are on distributed switch, then the backup job create task grays out forever. 

4. If the entire environment is migrated to standard switch, the working goes back to normal. 

So all in short, VDP 6.1 on a vDS environment has issues. Now, since migrating your entire networking to standard switch is obviously not a feasible or recommended task, there is a hot-patch released to fix this. 

Now, I am sharing these steps here along with the patch is solely because, I have distributed the patch to every customer who had opened a case with us to get this fixed along with the steps, so why not share it here for ease of access. 

Before we get to the resolution, this is what was noticed in the virgo logs in vCenter when the connect option was clicked when the VDP appliance was on a distributed switch.

[2016-04-11 18:36:45.604] [INFO ] http-bio-9443-exec-3 System.out [BlazeDS]Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'.
[2016-04-11 18:36:45.604] [INFO ] http-bio-9443-exec-3 System.out flex.messaging.MessageException: Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'. Type 'com.vmware.vim.binding.vim.dvs.PortConnection' not found.

[2016-04-11 18:36:45.608] [INFO ] http-bio-9443-exec-3 System.out [BlazeDS]Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'.
[2016-04-11 18:36:45.608] [INFO ] http-bio-9443-exec-3 System.out flex.messaging.MessageException: Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'. Type 'com.vmware.vim.binding.vim.dvs.PortConnection' not found.

[2016-04-11 18:36:45.612] [INFO ] http-bio-9443-exec-3 System.out [BlazeDS]Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'.
[2016-04-11 18:36:45.612] [INFO ] http-bio-9443-exec-3 System.out flex.messaging.MessageException: Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'. Type 'com.vmware.vim.binding.vim.dvs.PortConnection' not found.

[2016-04-11 18:36:45.616] [INFO ] http-bio-9443-exec-3 System.out [BlazeDS]Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'.
[2016-04-11 18:36:45.616] [INFO ] http-bio-9443-exec-3 System.out flex.messaging.MessageException: Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'. Type 'com.vmware.vim.binding.vim.dvs.PortConnection' not found.

So the resolution:

1. Download the vdr-ui-war-6.1.2.war file from this link here.
2. Determine the version of vCenter and navigate to one of the following directories accordingly:

5.5 vCenter
Windows:

C:\ProgramData\VMware\vSphere Web Client\vc-packages\vsphere-client-serenity\com.vmware.vdp2-<version>\plugins

Appliance:

var/lib/vmware/vsphere-client/vc-packages/vsphere-client-serenity/com.vmware.vdp2-<version>/plugins

6.0 vCenter
Windows:

C:\ProgramData\VMware\vCenterServer\cfg\vsphere-client\vc-packages\vsphere-client-serenity\com.vmware.vdp2-6.1.*\plugins

Appliance:

etc/vmware/vsphere-client/vc-packages/vsphere-client-serenity/com.vmware.vdp2-6.1.*/plugins/

3. Here you can see two files, a .jar file and a .war file. 
4. Rename the existing .war file to vdr-ui-war-6.1.x.war_old (where x is the version of your plugin)
5. Copy the patched .war file in the attachment, and paste it, named accordingly into this folder. 
6. In vCenter appliance 6.0, I had to perform couple of additional steps (Windows vCenter does not require these steps). The older jar and war file were having the following permissions:

Owner: vsphere-client
Group: Users

7. The applied patch has Owner and Group as root. Change this accordingly. 
You can do this easily by opening a WinSCP connection to vCenter > Right click the File > Properties and these two options will be available here. 
8. Restart the web client service. 

Windows, you can find this service in services.msc

Appliance

service vsphere-client stop

service vsphere-client start

9. Login to web client > Connect, and now we should be able to successfully connect to the appliance from web client. 

That's pretty much it. 
If something does not work, comment below! And always take care while patching.

Update:
If the connect still fails, then discard the old renamed .war file, restart the web client service and try connecting again.

Thank you!

VDP Backup Job Not Appearing In Recent Tasks

This is going to be a small article for few issues that I worked on recently. When you login to vSphere Web Client and run a backup job, the task progress does not show up under the "Recent Tasks" pane. When you login to vSphere Windows client, you can see the Create Snapshot and VDP backup job task. And in Web Client, when you click the Tasks section on the left hand side, you can see this task. 

This was the situation when my VDP user (The user with which I configured my VDP to vCenter) was different to the user that I had currently logged into web client with. 

The VDP user that I have configured is vsphere.local\Suhas and the user that I was logged in was vsphere.local\Administrator (Single Sign On user)

You cans see the user with which VDP is registered below:

Next, we will login to web client with the user that the VDP was registered to vCenter with, which is, vsphere.local\Suhas. And once logged in, I started another backup job and this time I can see the task in the "Recent tasks" pane. See screenshot below:

The resolution is simple, really simple. When you are logged in with a different user when compared to the user with which the VDP is registered with, Under the Recent Tasks pane, you have a drop-down which reads, My Tasks. Click this drop-down and select All Tasks, now you can see all the VDP backup tasks. See the screenshot below for the different user (SSO user)

That's it, really!

VDP File Level Recovery Client

So, you would be familiar with restoring virtual machines with VDP which is done from the Restore tab of the VDP appliance from the Web Client GUI. However, there are cases, where the necessity is to restore only certain files or folders and not the entire virtual machine or virtual disk. In this case, instead of using disk restore or virtual machine restore we perform File Level restore. 

Like your VDP configure page, FLR also has a client. The FLR client has to be accessed on the virtual machine where you want to perform the file restores. The URL for this would be:

https://VDP-IP:8543/flr

The login screen would look something like this:

This is the simple login page and there is an option for advanced login as well. We will have a look at the advanced login a little later in this article. 

For the simple login, you will have to provide the Windows machine's local administrator credentials. Now, for a small demo, on the machine where I have accessed the FLR, I had a file under "Desktop" directory called "Critical data" which had some data called "Critical Text". To give a little more background, once this file was created, a regular backup was executed for this virtual machine. The backup completed successfully. However, some hours later the file was lost. Now, we will be restoring that file using the FLR client. 

So I have logged in with my local admin credentials to the FLR client and this is the first screen I would see after a successful login. 

I have just one restore point created for this virtual machine since only one backup was taken. If you have multiple restore points after a set of backups, you can choose the restore point that you think is the best and select the option called Mount. 

You will then see this directory view. I will expand the drive and folders until, the necessary file is located. So my file was created under C > Users > Administrator > Desktop. Here the critical data text file is seen.

I will select only this file to restore and select the "Restore selected files" option. Upon selecting that you will have to provide a location as to where you want to restore the file. I will select the destination where it was originally residing. If you want to specify a different destination, you can do the same as well here. Click the Restore option once the directory is decided. 

You will receive a generic prompt stating that the operation might be time consuming. Select Yes to start the restore. 

Click Monitor Resources tab to check the restore status. Here the progress is not shown, and the only view available is as the one seen below:

From the vSphere client you can monitor the actual status of the restore.

Once the restore is completed successfully you can go to that directory and verify the file is restored successfully. 

The next option is restoring using the advanced login method. The advanced login method is used when, the machine where you actually want to restore the file has no access. In this case, with the advanced login, you will provide the credentials of the vCenter along with local credentials of the machine. 

The login would something like below:

Here the restore point screen looks a bit different. Instead of seeing only the restore point of the virtual machine, you will be seeing all the restore points for the virtual machines which were backed up.

Click the drop down for the virtual machine required, or you can also filter by name to locate it quickly. Click the required restore point from the list after selecting the drop-down and click mount. 

The remaining process is going to be same, you can restore the file to the same location or you can restore the file to a different location. 

One more thing to note:

Let's say, initially you had a text file with contents ABC. Then you took a backup of the virtual machine. After the backup, you made changes to the text file so your contents are now ABCDE. Now, if you restore this file on the same directory where the original file exists then the new contents of the file will be over written with the contents present in the restored file. So make sure that restore is done appropriately. 

That's it!

To know more about limitations of FLR you can click this link here to read the VDP admin guide. (See page 152)