Unable To Configure VDP To vCenter - Unable to find this VDP in the vCenter inventory

So, you might run into issues where you are unable to configure VDP to vCenter and you run into this error.

Unable to find this VDP in the vCenter inventory

In the vdr-configure.log you will notice the following. Again, for all issues with vdp-configure page refer the vdr-configure.log

2017-04-10 10:41:13,365 WARN  [http-nio-8543-exec-2]-vi.VCenterServiceImpl: No VCenter found in MC root domain
2017-04-10 10:41:13,365 INFO  [http-nio-8543-exec-2]-reconfig.VcenterConfigurationImpl: Failed to locate vCenter Client in Avamar, reconfiguration is required
2017-04-10 10:41:13,365 INFO  [http-nio-8543-exec-2]-sso.VmwareSsoServiceImpl: Getting SSL certificates for https://psc-prod:7444/lookupservice/sdk
2017-04-10 10:41:13,715 INFO  [http-nio-8543-exec-2]-services.VcenterConnectionTestService: Finished vCenter Connection test with result:
                <?xml version="1.0"?><vCenter><certValid>true</certValid><connection>true</connection><userAuthorized>true</userAuthorized><ave_in_vcenter>false</ave_in_vcenter><switch_needed>true<
/switch_needed><persistent_mode>true</persistent_mode><ssoValid>true</ssoValid><httpPortValid>true</httpPortValid></vCenter>

2017-04-10 10:41:13,025 WARN  [http-nio-8543-exec-2]-vi.VCenterServiceImpl: Failed to get root domain from MC
2017-04-10 10:41:13,025 WARN  [http-nio-8543-exec-2]-vi.VCenterServiceImpl: No VCenter found in MC root domain
2017-04-10 10:41:13,025 INFO  [http-nio-8543-exec-2]-vi.ViJavaServiceInstanceProviderImpl: visdkUrl = https://vc-prod:443/sdk
2017-04-10 10:41:13,337 INFO  [http-nio-8543-exec-2]-util.UserValidationUtil: vCenter user has sufficient privileges to run VDP.
2017-04-10 10:41:13,339 INFO  [http-nio-8543-exec-2]-network.NetworkInfoApi: Found IP Address: [10.116.189.178] link local? [false], site local? [true], loopback? [false]
2017-04-10 10:41:13,339 INFO  [http-nio-8543-exec-2]-network.NetworkInfoApi: Found IP Address: 10.116.189.178

2017-04-10 10:41:13,353 ERROR [http-nio-8543-exec-2]-vi.ViJavaAccess: getPoweredOnVmByIpAddr(): Cannot determine appropriate powered on AVE virtual machine with IP Address [10.x.x.x] since there exist many of them (2): type=VirtualMachine name=vdp-vm mor-id=vm-208, type=VirtualMachine name=Windows-Jump mor-id=vm-148

So in this case, 10.x.x.x is the IP of my VDP machine and there is a duplicate IP used by another VM in the vCenter and this is Windows-Jump. If this is the case, determine if you can remove the duplicate IP or change the IP of the VDP appliance. The configuration test should then complete without issues. 

Hope this helps.

Farewell vSphere Data Protection - End of Availability.

On April 5, VMware announced the end of vSphere Data Protection. vSphere 6.5 would be the last release to support VDP. Which means post this, you will need to migrate to third party backup.

The EOA details can be found in this link here:

http://www.vmware.com/products/vsphere/data-protection.html

The EOA KB article is published here:

https://kb.vmware.com/kb/2149614

" On April 5th, 2017, VMware announced the End of Availability (EOA) of the VMware vSphere Data Protection (VDP) product.

VMware vSphere 6.5 is the last release to include vSphere Data Protection and future vSphere releases will no longer include this product. We have received feedback that customers are looking to consolidate their backup and recovery solutions in support of their overall software-defined data center (SDDC) efforts. As a result, we are focusing our investments on vSphere Storage APIs – Data Protection to further strengthen the vSphere backup partner ecosystem that provides you with a choice of solution providers.
  
All existing vSphere Data Protection installations with active Support and Subscription (SnS) will continue to be supported until their End of General Support (EOGS) date. The EOGS dates for vSphere Data Protection are published on the VMware Lifecycle Product Matrix under the dates listed for different versions. After the EOA date, you can continue using your existing installations until your EOGS dates.

VMware supports a wide ecosystem of backup solutions that integrate with vSphere and vCenter using vSphere Storage APIs – Data Protection framework. You can use any data protection products that are based on this framework. 

Beginning today, Dell EMC is offering you a complimentary migration to the more robust and scalable Dell EMC Avamar Virtual Edition. VMware vSphere Data Protection is based on Dell EMC Avamar Virtual Edition, a key solution for protecting and recovering workloads across the SDDC. To learn more about this offer please go to the Dell EMC website.

If you have additional questions please contact your VMware Sales Representative or read the FAQ document "

However, the Support for VDP will continue to follow as per VMware SnS agreement from this link:

http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/support/product-lifecycle-matrix.pdf

Dell EMC will provide an offer to migrate VDP to AVE (Avamar Virtual Edition) here:

http://dellemc.com/vdpeoa

Any questions on the migration, refer the below FAQ:

http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsphere/vmw-vdp-eoa-faqs.pdf

I will continue to post articles on VDP and answer your questions as long as I am supporting it. I will be exploring more into the vRealize Suite from today with vRealize Operations to begin with. 

Comment to leave your thoughts. 

Well, you never know what you got until it's gone. 

Unable To Start Backup Scheduler In VDP 6.x

You might come across issues, where backup scheduler does not start when you try it from the vdp-configure page or the command line using dpnctl start sched. It fails with:

2017/03/22-18:58:53 dpnctl: ERROR: error return from "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mccli mcs resume-scheduler" - exit status 1

And the dpnctl.log will have the following:

2017/03/22-18:58:53 - - - - - - - - - - - - - - - BEGIN
2017/03/22-18:58:53 1,22631,Server has reached the capacity health check limit.
2017/03/22-18:58:53 Attribute Value
2017/03/22-18:58:53 --------- -------------------------------------------------------------------------------
2017/03/22-18:58:53 error     Cannot enable scheduler until health check limit reached event is acknowledged.
2017/03/22-18:58:53
2017/03/22-18:58:53 - - - - - - - - - - - - - - - END
2017/03/22-18:58:53 dpnctl: ERROR: error return from "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mccli mcs resume-scheduler" - exit status 1

If you run the below command you can see there are quite a few unacknowledged alarm that speaks about health check events not being acknowledged.

# mccli event show --unack=true | grep "22631"

1340224 2017-03-22 13:58:53 CDT WARNING 22631 SYSTEM   PROCESS  /      Server has reached the capacity health check limit.
1340189 2017-03-22 13:58:01 CDT WARNING 22631 SYSTEM   PROCESS  /      Server has reached the capacity health check limit.

To resolve this, acknowledge these events using the below command:

# mccli event ack --include=22631

Post this start the schedule either from GUI or command line using dpnctl start sched. 

Hope this helps.

Cowsay For Linux SSH Login

Cowsay has been around for quite a while now, but I came across it recently. I wanted to have a more interesting login for couple of data protection VMs and other CentOS boxes. If you follow this blog, you will know my domain is happycow.local, as the name "HappyCow" is quite fascinating, also it is my GamerTag on GTA5 (Hehe!).

Cowsay came to the rescue here to get this up and running in few steps. First, I had to get the cowsay package. You can download the package from here. SSH into your Linux box and have this package copied over.

Unzip the tar file by:

# tar -zxvf cowsay_3.03+dfsg2.orig.tar.gz

Post this, get into the directory cowsay-3.03+dfsg2 and run the installation script

# sh install.sh

Post this, create the below file:

# vi ~/.ssh/rc

Paste the content you want here for SSH login. My content was:

#!/bin/bash
clear
echo -e "Welcome to VDP \n If it is broken, redeploy" | cowsay
echo -e "\nYour system is been up for $(uptime | cut -d ' ' -f 4,5,6,7)"

Provide chmod u+x to rc file and then restart the sshd service

# service sshd restart

Log back into the terminal and you will see the "Zen-Cow" greeting you.

Looks fun!

Automating Backup Cancellation From Command Line

This script allows you to mass cancel active backup jobs from command line of vSphere Data Protection Appliance.

#!/bin/bash
# This script cancels all active backup jobs from the command line
value=$(mccli activity show --active | cut -c1-16 | sed -e '1,3d')
if [ -z "$value" ]
then
echo "No active job to cancel"
else
for p in $value
do
mccli activity cancel --id=$p
done
fi

If you would like to cancel a set of backup jobs, like 13 jobs out of 20 running jobs, then you need to add those Job ID's to a file and then run the script to pull inputs from that file

#!/bin/bash
# This script cancels jobs from IDs provided in the id.txt file
while read p; do
mccli activity cancel --id=$p
done <id.txt

This script can be modified for other backup states like waiting-client. Just Grep, and cut, and remove the first three rows and feed the job ID's to a loop.

A much more interactive script to cancel "Active" "Waiting-Queued" and "Waiting-Client" jobs.

#!/bin/bash
# This block is for help parameters.
usage()
{
cat << EOF

Below are the available fields

OPTIONS:
   -h      Help
   -a      Active Job
   -w      Waiting Job
EOF
}
# This block saves status of active/waiting-client/waiting-queued backups
value=$(mccli activity show --active | cut -d ' ' -f 1 | sed -e '1,3d')
value_client=$(mccli activity show | grep -i "Waiting-Client" | cut -d ' ' -f 1)
value_queued=$(mccli activity show | grep -i "Waiting-Queued" | cut -d ' ' -f 1)

# This block does a flag input
while getopts "haw" option
do
        case $option in
                a)

        if [ -z $value ]
        then
                printf "No active jobs to cancel\n"
        else

                printf "Cancelling active jobs\n"
                for i in $value
                do
                mccli activity cancel --id=$i
                done

        fi
                ;;
                w)
                if [ -z $value_client ]
                then
                        echo $value_client
                        printf "No jobs in waiting client state\n"
                else
                        printf "Cancelling waiting clients\n"
                        for i in $value_client
                        do
                                mccli activity cancel --id=$i
                        done
                fi
                if [ -z $value_queued ]
                then
                        printf "No jobs in waiting queued state\n"
                else
                        printf "Cancelling queued clients\n"
                        for i in $value_queued
                        do
                                mccli activity cancel --id=$i
                        done
                fi
                ;;
h)
usage
;;
?)
printf "type -h for list\n"
;;
esac
done

Chmod a+x to the file for execute. Hope this helps!

Deploying vSphere Data Protection From govc CLI

In vSphere 6.5 GA there have been a lot of reported instances where we are unable to deploy any ova template. In this article, I will be talking in specific to vSphere Data Protection. As you know, vSphere 6.5 supports only 6.1.3 of VDP. If you try to deploy this via the Web Client, you will run into issues stating "Use a 6.5 version of web client". The workaround would be to use the OVF-tool to have this appliance deployed on a vCenter. Personally, I find ovf tool to be a bit challenging for first time users. A simple way would be to use the govc CLI to have this template deployed. William Lam has this written about this in a greater detail as to what this tool is all about and you can read it here

Here, I am using a CentOS machine to stage the deployment. 

1. The first step would be to download the appropriate govc binary. To access the list, you can visit the gitHub link here. Once you have the required binary listed out in that link, run the below command to download the binary on to your Linux box.

# curl -L https://github.com/vmware/govmomi/releases/download/v0.5.0/govc_linux_386.gz | gunzip -d > /usr/local/bin/govc

I am using govc_linux_386.gz as it is compatible with my CentOS, if you are using a different distro or a windows based system, choose accordingly.

You should see a below task to indicate the download is in progress:

[root@centOS /]# curl -L https://github.com/vmware/govmomi/releases/download/v0.5.0/govc_linux_386.gz | gunzip -d > /usr/local/bin/govc

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 6988k  100 6988k    0     0   713k      0  0:00:09  0:00:09 --:--:-- 1319k

2. Once the download is done, provide execute permissions to this binary. Run this command:

# chmod +x /usr/local/bin/govc

Note: The download can be done to any required directory.

3. Verify the download is successful and govc is working by checking for the version:

# govc version

You should see:

govc v0.5.0

4. We will have to set few environment variables to define which host, storage and network this VDP virtual machine should be deployed on. 

export GOVC_INSECURE=1

export GOVC_URL=<Specify-ESXi-or-VC-FQDN>

export GOVC_USERNAME=<User-login-for-the-above>

export GOVC_PASSWORD=<Your-Password>

export GOVC_DATASTORE=<Datastore-Name>

export GOVC_NETWORK=<Network-Portgroup>

export GOVC_RESOURCE_POOL=<Resource-pool-if-you-have-one>

5. Next, we will have to create a json specification file to provide the details of the VDP appliance. Run the below command to view the specification:

# govc import.spec /vSphereDataProtection-6.1.3.ova | python -m json.tool

You will notice the below:

6. Redirect this output to a file so that we can edit and provide the necessary details. Run this command:

# govc import.spec /vSphereDataProtection-6.1.3.ova | python -m json.tool > vdp.json

7. Open the file in a vi editor and enter the details for networking. Remove the first line which says "Deployment": "small", If this is not done, your deployment will fail with:

" govc: ServerFaultCode: A specified parameter was not correct: cisp.deploymentOption " 

You should have something similar post you edit the file:

Save the file.

8. Lastly, we will be deploying the ova template using the import.ova function. And during this deployment we will make use of the json file we created which has the networking details and the environment variables where we specified the location for ova deployment. The command would be:

# govc import.ova -options=vdp.json /vSphereDataProtection-6.1.3.ova

You should now see a progress bar:

[root@centOS ~]# govc import.ova -options=vdp.json /vSphereDataProtection-6.1.3.ova

[06-03-17 14:14:57] Warning: Line 139: Invalid value 'Isolated Network' for element 'Connection'.

[06-03-17 14:15:03] Uploading vSphereDataProtection-0.0TB-disk1.vmdk... (1%, 5.1MiB/s)

9. Post this you can power on your VDP appliance and begin the configuration. 

Hope this helps. 

VDP Backup Fails With The Error "Failed To Attach Disks"

Earlier, we had seen a compatibility issue with VDP 6.1.3 on an ESXi 5.1 environment, where the backup used to fail with the message "Failed to attach disks". More about this can be read here
However, this is a very generic message and can mean different if we are running VDP on a compatible version.

In this case, the VDP was 6.1.3 on a 6.0 vSphere environment and the backup used to fail only when an external proxy was deployed. If the external proxy was discarded the backups utilized the internal VDP proxy and completed successfully.

With the external proxy, the logs are on the proxy machine under /usr/local/avamarclient/var
The backup job logs had the following entry:

2017-03-02T16:13:18.762Z avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_PrepareForAccess: Disable Storage VMotion failed. Error 18000 (Cannot connect to the host) (fault (null), type GVmomiFaultInvalidResponse, reason: (none given), translated to 18000) at 4259.

2017-03-02T15:46:19.092Z avvcbimage Info <16041>: VDDK:VixDiskLibVim: Error 18000 (listener error GVmomiFaultInvalidResponse).

2017-03-02T15:46:19.092Z avvcbimage Warning <16041>: VDDK:VixDiskLibVim: Login failure. Callback error 18000 at 2444.

2017-03-02T15:46:19.092Z avvcbimage Info <16041>: VDDK:VixDiskLibVim: Failed to find the VM. Error 18000 at 2516.

2017-03-02T15:46:19.093Z avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVim_FreeNfcTicket: Free NFC ticket.

2017-03-02T15:46:19.093Z avvcbimage Info <16041>: VDDK:VixDiskLib: Error occurred when obtaining NFC ticket for [Datastore_A] Test_VM/Test_VM.vmdk. Error 18000 (Cannot connect to the host) (fault (null), type GVmomiFaultInvalidResponse, reason: (none given), translated to 18000) at 2173.

2017-03-02T15:46:19.093Z avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_OpenEx: Cannot open disk [Datastore_A] Test_VM/Test_VM.vmdk. Error 18000 (Cannot connect to the host) at 4964.

2017-03-02T15:46:19.093Z avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_Open: Cannot open disk [Datastore_A] Test_VM/Test_VM.vmdk. Error 18000 (Cannot connect to the host) at 5002.

2017-03-02T15:46:19.093Z avvcbimage Error <0000>: [IMG0008] Failed to connect to virtual disk [Datastore_A] Test_VM/Test_VM.vmdk (18000) (18000) Cannot connect to the host

In the mcserver.log, the following was noted:

WARNING: com.avamar.mc.sdk10.McsFaultMsgException: E10055: Attempt to connect to virtual disk failed.
at com.avamar.mc.sdk10.util.McsBindingUtils.createMcsFaultMsg(McsBindingUtils.java:35)
at com.avamar.mc.sdk10.util.McsBindingUtils.createMcsFault(McsBindingUtils.java:59)
at com.avamar.mc.sdk10.util.McsBindingUtils.createMcsFault(McsBindingUtils.java:63)
at com.avamar.mc.sdk10.mo.JobMO.monitorJobs(JobMO.java:299)
at com.avamar.mc.sdk10.mo.GroupMO.backupGroup_Task(GroupMO.java:258)
at com.avamar.mc.sdk10.mo.GroupMO.execute(GroupMO.java:231)
at com.avamar.mc.sdk10.async.AsyncTaskSlip.run(AsyncTaskSlip.java:77)

The cause of this is due to an Ipv6 AAAA record. VDP does not support a dual stack networking and needs to have either IPv4 settings or IPv6 settings.

Resolution:
1. Login to the external proxy machine using root credentials
2. Run the below command to test DNS resolution:

# nslookup -q=any <vcenter-fqdn>

An ideal output should be as follows:

root@vdp-dest:~/#: nslookup -q=any vcenter-prod.happycow.local
Server:         10.109.10.140
Address:        10.109.10.140#53

Name:   vcenter-prod.happycow.local
Address: 10.109.10.142

But, if you see the below output, then you have an IPv6 AAAA record as well:

root@vdp-dest:~/#: nslookup -q=any vcenter-prod.happycow.local
Server:         10.109.10.140
Address:        10.109.10.140#53

Name:   vcenter-prod.happycow.local

Address: 10.109.10.142

vcenter-prod.happycow.local   has AAAA address ::9180:aca7:85e7:623d

3. Run the below command to set IPv4 precedence over IPv6:

echo "precedence ::ffff:0:0/96  100" >> /etc/gai.conf

4. Restart the avagent service using the below command:

# service avagent-vmware restart

Post this, the backups should work successfully. If the Ipv6 entry is not displayed in the nslookup and the backup still fails, then please raise a support request with VMware.