VDP 6.1.4 Backup Failures For ESXi 5.1

When you backup a VM residing on ESXi 5.1 which does not share the storage with the ESXi hosting the VDP appliance, the backup will use nbd / nbdssl protocol. With VDP 6.1.4 these backups will fail.

In the backup log you will notice the following:

2017-06-14T09:21:07.869+05:00 avvcbimage Info <14700>: submitting pax container in-use block extents:
2017-06-14T09:21:07.899+05:00 avvcbimage Warning <16041>: VDDK:[NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 46
 2017-06-14T09:21:07.900+05:00 avvcbimage Info <16041>: VDDK:DISKLIB-LIB   : RWv failed ioId: #1 (290) (34) .
 2017-06-14T09:21:07.900+05:00 avvcbimage Info <16041>: VDDK:VixDiskLib: Detected DiskLib error 290 (NBD_ERR_GENERIC).
2017-06-14T09:21:07.900+05:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_Read: Read 128 sectors at 0 failed. Error 1 (Unknown error) (DiskLib error 290: NBD_ERR_GENERIC) at 5178.

2017-06-14T09:21:07.900+05:00 avvcbimage Error <0000>: [IMG0008] VixDiskLib_Read() at offset 0 length 128 returned 1 (1) Unknown error
2017-06-14T09:21:07.900+05:00 avvcbimage Error <0000>: [IMG0008] VixDiskLib_Read() ([Prod-Datastore] VM-Prod/VM-Prod.vmdk) at offset 0 length 128 sectors returned (1) (1) Unknown error
2017-06-14T09:21:07.900+05:00 avvcbimage Info <9772>: Starting graceful (staged) termination, VixDiskLib_Read returned an error (wrap-up stage)
2017-06-14T09:21:07.901+05:00 avvcbimage Info <40654>: isExitOK()=157
2017-06-14T09:21:07.901+05:00 avvcbimage Info <16022>: Cancel detected(miscellaneous error), isExitOK(0).
2017-06-14T09:21:07.901+05:00 avvcbimage Info <9746>: bytes submitted was 0

You might also notice a back trace:

2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  INTERNAL ERROR: <0001> assert error (uapp::staging().isExitCancel() || list == __null || skipped_ntfs_chunks), /local/jenkins/workspace/client_CuttyS
ark_SLES11-64/abs2/work/src/avclient/filepipe.cpp line 101
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000006b93b1
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000006ba127
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000006ba292
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000006ba3fe
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000006ba87d
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 000000000060cb74
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 000000000060d67d
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000006d9379
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000006dc9a1
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000006dc200
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000006d9379
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000005e5af1
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 000000000056455a
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 0000000000442c07
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000004844d5
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00000000004844f9
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 000000000049a0bd
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 000000000070ac4c
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00007f8a8d282806
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread]  | 00007f8a8bec79bd

This is because, in the newer release of VDP for backup over network, they have added couple of new compression algorithms. Zlib, FastLZ, SkipZ in the VDDK module. And ESXi 5.1 does not support this.

You could try making the VMs storage on ESXi 5.1 available to the VDP's ESXi host so a hotadd protocol can be used.

But the bottom line is, Please! Upgrade your ESXi from 5.1 (This is officially unsupported for 6.1.3 and 6.1.4 VDP)

Hope this helps.

Deploying vSphere Replication Using OVF Tool

If you ever run into vSphere Replication Appliance deployment issues from web client (Specially in Pre U1 release of vCenter 6.5) you can use the OVF tool to deploy this appliance. Here is a quick walk-through of how to get this done.

1. Download the OVF tool using the below link
Download: https://my.vmware.com/web/vmware/details?downloadGroup=OVFTOOL400&productId=353

2. Install the OVF tool by running the exe file. The installation is pretty straight forward. As simple as calculating 5x5

3. Open the Command Prompt in administrator mode and change your directory to the install location of OVF tool. By default, this should be:

C:\Program Files\VMware\VMware OVF Tool

4. Below is the syntax for OVF tool deployment:

ovftool.exe --acceptAllEulas -ds="DATASTORE_NAME" -n="SPECIFY VRMS NAME" --net:"Management Network"="PORT GROUP NAME" --prop:"password"="VRMS ROOT PASSWORD" --prop:"ntpserver"="NTP SERVER IP OR FQDN" --prop:"vami.ip0.vSphere_Replication_Appliance"="SPECIFY VRMS SERVER IP" --vService:installation=com.vmware.vim.vsm:extension_vservice <PATH>\vSphere_Replication_OVF10.ovf vi://"administrator@vsphere.local":<SSO-PASSWORD>@VCENTER IP/?ip=HOST IP

Below is the sample command from my environment:

ovftool.exe --acceptAllEulas -ds="Local_Storage_1_39" -n="VR-OVF" --net:"Management Network"="VM Network 2" --prop:"password"="HappyCow123!" --prop:"ntpserver"="10.x.x.x" --prop:"vami.ip0.vSphere_Replication_Appliance"="10.x.x.x" --vService:installation=com.vmware.vim.vsm:extension_vservice E:\bin\vSphere_Replication_OVF10.ovf vi://"administrator@vmware.local":HappyCow123!@10.x.x.x/?ip=10.x.x.x

The switches:
-ds is for datastore
-n is to specify display name for the vR appliance (Not the host name, but the VM inventory name)
--net is to specify the port group for vR
--prop:"password" is vR root password
--prop:"ntpserver" is for your time server

The path would be the location of the OVF10.ovf file. If you downloaded like say 6.1.1 ISO and mounted it, then it would be the <CD-ROM-Drive>\bin\vSphere_Replication_OVF10.ovf

Always use SSO username for deployment and make sure it is within the double quotes, else you may run into issue where the '@' is not recognized

You should see a similar output once you execute the command:

Opening OVF source: //:@jump.happycow.local:80\bin\vSphere_Replication_O
The manifest validates
Source is signed and the certificate validates
Opening VI target: vi://administrator%40vmware.local@10.x.x.x:443/
Deploying to VI: vi://administrator%40vmware.local@10.x.x.x:443/
Transfer Completed
Completed successfully

You will also see a progress percentage along with a recent task for OVF deployment in the vSphere Client to monitor.

Hope this was helpful.

Backup vSphere Data Protection Using vSphere Data Protection? Really!

Some of you might have tried to backup VDP using a VDP appliance and would have failed to get this VM added to a backup job. The admin guide states that you cannot backup VDP using a VDP appliance. This is mainly because the data0? partitions of your data protection appliance are Independent persistent XFS drives which do not support snapshot feature. Yet, backing up the OS drive is some of us look forward to, right? Your OS drives contains your backup job, replication job and yeah, "logs", which no one needs. 

Though not an officially supported way, here is a sneaky way to work around this. At your own risk!

Why does the VDP VM not list in the VM section when you create a backup job? 

Good question. If you select the VDP VM in the vSphere Client inventory and then hit the summary tab, under Annotations section, you will notice you have "VMware vSphere Data Protection Appliance 6.1" under Notes

If any VM has this text in the Notes section, then it will not be listed in the inventory list while creating a backup Job. Go ahead, spin up a test VM, and then try creating a backup Job, you will notice your test VM. Then add this Notes section and you will notice this VM no longer shows up in the Inventory list during Job creation.

So you guessed it right, Go ahead and remove this Notes section from your VDP appliance and you will now see the VDP listed under the job create wizard. Make sure the job you create is a disk level job and not a complete VM level backup.

Note while doing this:

You will need a second VDP to backup the first one. This would be because, if you remove the Notes section for the VDP, the appliance would no longer be listed in the plugin drop-down. In my case, I had two VDP and to backup OS disk of VDP 1 I did this:

> Removed the Notes section for VDP1
> Connected to VDP2 and added VDP1's OS disk to a job
> Backed up this disk
> Re-added the Notes section back to VDP1 and it was listed back in the plugin.  

Hope this helps. In some way. Heh!

Unable To Send Test Email In vSphere Data Protection 6.x

Once you configure email settings in VDP, the first thing we do is send a test email to verify the configuration is good. The test email content should be something like:

This email has been generated by VMware vSphere Data Protection to test your email report settings.

However, you might come across an issue, where the test email does not trigger and you will see the following error in the web client:

In the vdr-server.log you can notice the following:

2017-06-07 01:46:27,636 ERROR [http-nio-8543-exec-9]-email.EmailBase: On try 0to send the email, recieved a connection error from the external email server:Could not connect to SMTP host: email.vmware.com, port:
 587
2017-06-07 01:46:27,636 ERROR [http-nio-8543-exec-9]-services.EmailConfigServiceWS: Unable to trigger test email
javax.mail.MessagingException: Could not connect to SMTP host: email.vmware.com, port: 587;
  nested exception is:
        java.net.ConnectException: Connection timed out (Connection timed out)
        at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1282)
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:370)
        at javax.mail.Service.connect(Service.java:297)
        at javax.mail.Service.connect(Service.java:156)
        at javax.mail.Service.connect(Service.java:105)
        at javax.mail.Transport.send0(Transport.java:168)
        at javax.mail.Transport.send(Transport.java:98)
        at com.emc.vdp2.common.email.EmailBase.sendReport(EmailBase.java:174)
        at com.emc.vdp2.common.email.EmailBase.sendReport(EmailBase.java:63)
        at com.emc.vdp2.email.EmailSummaryReport.sendTestEmail(EmailSummaryReport.java:129)
        at com.emc.vdp2.services.EmailConfigServiceWS.runEmailTest(EmailConfigServiceWS.java:93)
        at com.emc.vdp2.rest.ReportService.runEmailTest(ReportService.java:241)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)

Use SMTP port 25 to fix this issue.
So in the configuration of mail settings, in the field for Outgoing mail server, specify
<your mail sever > : 25

Note that there is <space> before and after the :
Save the settings and now the test email should work.

I believe port 587 is block on Firewall.base for VDP and explicitly mentioning 25 resolves this, as SMTP 25 is open on VDP.
You can netstat -ano | grep 25 and netstat -ano | grep 587

Hope this helps!

Unable To Send Email Report In VDP 6.1.4

This issue is applicable to 6.1.4 only. Here, you will be able to send a Test email, however, the scheduled email reports are not sent. In the vdr-server.log, you notice the following NullPointerException.

In the GUI you might see:

"Send email summary report error: Failed to send the email summary report. Reason: null"

vdr-server.log:

ERROR [http-nio-8543-exec-6]-services.EmailConfigServiceWS: Unable to trigger test email java.lang.NullPointerException
        at com.emc.vdp2.common.vi.ViJavaAccess.getVmInstanceUuidByIpAddress(ViJavaAccess.java:558)
        at com.emc.vdp2.email.VirtualMachineSummary.addDiskRequirements(VirtualMachineSummary.java:240)
        at com.emc.vdp2.email.VirtualMachineSummary.getReportNode(VirtualMachineSummary.java:222)
        at com.emc.vdp2.email.VirtualMachineSummary.getVirtualMachinesReport(VirtualMachineSummary.java:131)
        at com.emc.vdp2.email.EmailSummaryReport.createReport(EmailSummaryReport.java:347)
        at com.emc.vdp2.email.EmailSummaryReport.createReport(EmailSummaryReport.java:260)
        at com.emc.vdp2.email.EmailSummaryReport.createAndSendReport(EmailSummaryReport.java:142)
        at com.emc.vdp2.services.EmailConfigServiceWS.sendEmailReport(EmailConfigServiceWS.java:111)
        at com.emc.vdp2.rest.ReportService.sendEmailReport(ReportService.java:281)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)

To fix this:

1. Download the zip file from this link here.
2. The md5 hash for this is: 90a57acb1404cdff68181b6768d3b539 Verify if this is the same on the downloaded zip
3. Use WinScp and copy this file to the /root folder of VDP
4. From SSH of the VDP appliance stop the tomcat service using:

# emwebapp.sh --stop

5. Backup the original vdr-server.war file using this command:

# cp -p /usr/local/avamar/lib/vdr-server.war /root/vdr-server.war_ORIG

6. Copy the patched binary using this command:

# cp /root/vdr-server.war /usr/local/avamar/lib/vdr-server.war

Make sure the permission for this patched binary is same as the original file.

7. Remove the vdr-server* files from tomcat using:

# rm -R /usr/local/avamar-tomcat/webapps/vdr-server*

There will be two files under this webapps directory for vdr-server. Verify before removing and these will be recreated once tomcat is restarted.

8. Start the tomcat service using:

# emwebapp.sh --start

9. The vdr server initialization takes a while and then you should be able to connect to VDP in web client and test your email reporting.

Or, you can download the script from github and run it on VDP to fix this automatically. Download vdp614-email-fix.sh
https://github.com/happycow92/shellscripts

If this still does not work, open a case with VMware support. Hope this helps!

Bash Script To Extend Retention/Expiry Date Of A Restore Point In vSphere Data Protection 6.x

When you create a backup job, you get an option to specify how long should the backed up data be retained in the VDP appliance. Once the client is backed up, changing this expiry date is quite a challenge. You can use the below available script to perform this task.

Note that retention can be only increased using this script and cannot be decreased.

1. Place the script in /root folder of VDP and run the script while the Run User is "root"
2. Provide execute permissions to this script

# chmod +x retention-extend.sh

3. Run the script as ./retention-extend

Test the script on a few test VMs and when comfortable you can move on to the remaining VMs. Please note, the script is not from VMware and is self developed. Run this at your own risk. (Should not affect anything on the appliance, but still. always a disclaimer!)

Download the script from:
https://github.com/happycow92/re

View the source code:
https://github.com/happycow92/re/blob/master/retention-extend.sh

Hope this helps.

VDP 6.x FLR - Failed To Get Disks: The VMDK Filename Is Not Valid Or Present

In this article, we will discuss about FLR issue Specifically, how the FLR functionality breaks when a VDP IP/hostname reconfiguration is done. Thanks to one of our customer's who logged in a ticket with VMware for this issue. This would definitely be one of the most awaited workarounds.

So, in the vdp-configure page, under proxy you will have the proxy name as the hostname of the VDP appliance (If Internal proxy is being used) When a hostname change of VDP is done, this value does not get updated in addition to one of the proxy configuration files on which FLR is based. This causes FLR to give the error:

Failed to get disks: The VMDK filename is not valid or present. Verify that the proxy was correctly registered using the supported method in the documentation.

In the command line, you can run the below command to extract the name of the internal proxy:

# mccli client show --recursive=true | grep -i /clients | awk '{print $1}'

In my case, I had a VDP with name vdp-dest.happycow.local with an IP. I changed this to vdp-new-dest.happycow.local. Then when I checked "hostname" and the proxy name using the above command, both of them reflected the updated address.

Due to this, your backups and full image restores continue to work without issues. FLR uses axionfs.cmd file to pull server parameters.

This file is located under:

# cd /usr/local/avamar/var

If you vi this file, you will notice the --server parameter is still having the old FQDN.

--server=vdp-dest.happycow.local
--id=restoreonly
--password=1aa34f26dc161939f7452aa3302133075449750f01c02148814a4563bb0965f96a61cdf9bfc96114
--fuseoptions='-s -r -f -o allow_other,use_ino'
--primarybackupdir=by-number
--inactivity_timeout=172800
--encrypt=tls
--encrypt-strength=high

So you will have to make sure the right FQDN is reflected under the --server parameter. Once this is done restart the axionfs service using:

# service axionfs restart

Post this, you should be able to perform FLR. The internal proxy would not reflect with the new hostname even if a re-register of proxy is done, however this should not be an issue. 

Hope this helps.