Backup vSphere Data Protection Using vSphere Data Protection? Really!

Some of you might have tried to backup VDP using a VDP appliance and would have failed to get this VM added to a backup job. The admin guide states that you cannot backup VDP using a VDP appliance. This is mainly because the data0? partitions of your data protection appliance are Independent persistent XFS drives which do not support snapshot feature. Yet, backing up the OS drive is some of us look forward to, right? Your OS drives contains your backup job, replication job and yeah, "logs", which no one needs. 

Though not an officially supported way, here is a sneaky way to work around this. At your own risk!

Why does the VDP VM not list in the VM section when you create a backup job? 

Good question. If you select the VDP VM in the vSphere Client inventory and then hit the summary tab, under Annotations section, you will notice you have "VMware vSphere Data Protection Appliance 6.1" under Notes

If any VM has this text in the Notes section, then it will not be listed in the inventory list while creating a backup Job. Go ahead, spin up a test VM, and then try creating a backup Job, you will notice your test VM. Then add this Notes section and you will notice this VM no longer shows up in the Inventory list during Job creation.

So you guessed it right, Go ahead and remove this Notes section from your VDP appliance and you will now see the VDP listed under the job create wizard. Make sure the job you create is a disk level job and not a complete VM level backup.

Note while doing this:

You will need a second VDP to backup the first one. This would be because, if you remove the Notes section for the VDP, the appliance would no longer be listed in the plugin drop-down. In my case, I had two VDP and to backup OS disk of VDP 1 I did this:

> Removed the Notes section for VDP1
> Connected to VDP2 and added VDP1's OS disk to a job
> Backed up this disk
> Re-added the Notes section back to VDP1 and it was listed back in the plugin.  

Hope this helps. In some way. Heh!

Unable To Send Test Email In vSphere Data Protection 6.x

Once you configure email settings in VDP, the first thing we do is send a test email to verify the configuration is good. The test email content should be something like:

This email has been generated by VMware vSphere Data Protection to test your email report settings.

However, you might come across an issue, where the test email does not trigger and you will see the following error in the web client:

In the vdr-server.log you can notice the following:

2017-06-07 01:46:27,636 ERROR [http-nio-8543-exec-9]-email.EmailBase: On try 0to send the email, recieved a connection error from the external email server:Could not connect to SMTP host: email.vmware.com, port:
 587
2017-06-07 01:46:27,636 ERROR [http-nio-8543-exec-9]-services.EmailConfigServiceWS: Unable to trigger test email
javax.mail.MessagingException: Could not connect to SMTP host: email.vmware.com, port: 587;
  nested exception is:
        java.net.ConnectException: Connection timed out (Connection timed out)
        at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1282)
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:370)
        at javax.mail.Service.connect(Service.java:297)
        at javax.mail.Service.connect(Service.java:156)
        at javax.mail.Service.connect(Service.java:105)
        at javax.mail.Transport.send0(Transport.java:168)
        at javax.mail.Transport.send(Transport.java:98)
        at com.emc.vdp2.common.email.EmailBase.sendReport(EmailBase.java:174)
        at com.emc.vdp2.common.email.EmailBase.sendReport(EmailBase.java:63)
        at com.emc.vdp2.email.EmailSummaryReport.sendTestEmail(EmailSummaryReport.java:129)
        at com.emc.vdp2.services.EmailConfigServiceWS.runEmailTest(EmailConfigServiceWS.java:93)
        at com.emc.vdp2.rest.ReportService.runEmailTest(ReportService.java:241)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)

Use SMTP port 25 to fix this issue.
So in the configuration of mail settings, in the field for Outgoing mail server, specify
<your mail sever > : 25

Note that there is <space> before and after the :
Save the settings and now the test email should work.

I believe port 587 is block on Firewall.base for VDP and explicitly mentioning 25 resolves this, as SMTP 25 is open on VDP.
You can netstat -ano | grep 25 and netstat -ano | grep 587

Hope this helps!

Unable To Send Email Report In VDP 6.1.4

This issue is applicable to 6.1.4 only. Here, you will be able to send a Test email, however, the scheduled email reports are not sent. In the vdr-server.log, you notice the following NullPointerException.

In the GUI you might see:

"Send email summary report error: Failed to send the email summary report. Reason: null"

vdr-server.log:

ERROR [http-nio-8543-exec-6]-services.EmailConfigServiceWS: Unable to trigger test email java.lang.NullPointerException
        at com.emc.vdp2.common.vi.ViJavaAccess.getVmInstanceUuidByIpAddress(ViJavaAccess.java:558)
        at com.emc.vdp2.email.VirtualMachineSummary.addDiskRequirements(VirtualMachineSummary.java:240)
        at com.emc.vdp2.email.VirtualMachineSummary.getReportNode(VirtualMachineSummary.java:222)
        at com.emc.vdp2.email.VirtualMachineSummary.getVirtualMachinesReport(VirtualMachineSummary.java:131)
        at com.emc.vdp2.email.EmailSummaryReport.createReport(EmailSummaryReport.java:347)
        at com.emc.vdp2.email.EmailSummaryReport.createReport(EmailSummaryReport.java:260)
        at com.emc.vdp2.email.EmailSummaryReport.createAndSendReport(EmailSummaryReport.java:142)
        at com.emc.vdp2.services.EmailConfigServiceWS.sendEmailReport(EmailConfigServiceWS.java:111)
        at com.emc.vdp2.rest.ReportService.sendEmailReport(ReportService.java:281)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)

To fix this:

1. Download the zip file from this link here.
2. The md5 hash for this is: 90a57acb1404cdff68181b6768d3b539 Verify if this is the same on the downloaded zip
3. Use WinScp and copy this file to the /root folder of VDP
4. From SSH of the VDP appliance stop the tomcat service using:

# emwebapp.sh --stop

5. Backup the original vdr-server.war file using this command:

# cp -p /usr/local/avamar/lib/vdr-server.war /root/vdr-server.war_ORIG

6. Copy the patched binary using this command:

# cp /root/vdr-server.war /usr/local/avamar/lib/vdr-server.war

Make sure the permission for this patched binary is same as the original file.

7. Remove the vdr-server* files from tomcat using:

# rm -R /usr/local/avamar-tomcat/webapps/vdr-server*

There will be two files under this webapps directory for vdr-server. Verify before removing and these will be recreated once tomcat is restarted.

8. Start the tomcat service using:

# emwebapp.sh --start

9. The vdr server initialization takes a while and then you should be able to connect to VDP in web client and test your email reporting.

Or, you can download the script from github and run it on VDP to fix this automatically. Download vdp614-email-fix.sh
https://github.com/happycow92/shellscripts

If this still does not work, open a case with VMware support. Hope this helps!

Bash Script To Extend Retention/Expiry Date Of A Restore Point In vSphere Data Protection 6.x

When you create a backup job, you get an option to specify how long should the backed up data be retained in the VDP appliance. Once the client is backed up, changing this expiry date is quite a challenge. You can use the below available script to perform this task.

Note that retention can be only increased using this script and cannot be decreased.

1. Place the script in /root folder of VDP and run the script while the Run User is "root"
2. Provide execute permissions to this script

# chmod +x retention-extend.sh

3. Run the script as ./retention-extend

Test the script on a few test VMs and when comfortable you can move on to the remaining VMs. Please note, the script is not from VMware and is self developed. Run this at your own risk. (Should not affect anything on the appliance, but still. always a disclaimer!)

Download the script from:
https://github.com/happycow92/re

View the source code:
https://github.com/happycow92/re/blob/master/retention-extend.sh

Hope this helps.

VDP 6.x FLR - Failed To Get Disks: The VMDK Filename Is Not Valid Or Present

In this article, we will discuss about FLR issue Specifically, how the FLR functionality breaks when a VDP IP/hostname reconfiguration is done. Thanks to one of our customer's who logged in a ticket with VMware for this issue. This would definitely be one of the most awaited workarounds.

So, in the vdp-configure page, under proxy you will have the proxy name as the hostname of the VDP appliance (If Internal proxy is being used) When a hostname change of VDP is done, this value does not get updated in addition to one of the proxy configuration files on which FLR is based. This causes FLR to give the error:

Failed to get disks: The VMDK filename is not valid or present. Verify that the proxy was correctly registered using the supported method in the documentation.

In the command line, you can run the below command to extract the name of the internal proxy:

# mccli client show --recursive=true | grep -i /clients | awk '{print $1}'

In my case, I had a VDP with name vdp-dest.happycow.local with an IP. I changed this to vdp-new-dest.happycow.local. Then when I checked "hostname" and the proxy name using the above command, both of them reflected the updated address.

Due to this, your backups and full image restores continue to work without issues. FLR uses axionfs.cmd file to pull server parameters.

This file is located under:

# cd /usr/local/avamar/var

If you vi this file, you will notice the --server parameter is still having the old FQDN.

--server=vdp-dest.happycow.local
--id=restoreonly
--password=1aa34f26dc161939f7452aa3302133075449750f01c02148814a4563bb0965f96a61cdf9bfc96114
--fuseoptions='-s -r -f -o allow_other,use_ino'
--primarybackupdir=by-number
--inactivity_timeout=172800
--encrypt=tls
--encrypt-strength=high

So you will have to make sure the right FQDN is reflected under the --server parameter. Once this is done restart the axionfs service using:

# service axionfs restart

Post this, you should be able to perform FLR. The internal proxy would not reflect with the new hostname even if a re-register of proxy is done, however this should not be an issue. 

Hope this helps.

Unable To Install SQL/Exchange Backup Agent on vSphere Data Protection

When you upgrade your VDP from 5.8.x to/ot 6.0.x to/or 6.1.x you will need to uninstall the old backup agent, re download the new agent from the 6.1.x Configuration Tab and install it. However, this installation will fail with the below error:

This is because the client was not deactivated once the agent was uninstalled. Until you deactivate the client associated with the old plugin you will not be able to install the new one.

On the machine where you are installing the client, if you browse to C:\Program Files\avp\var you will have a log file called avagent.log and if you notice the error here, you will see:

2017-05-19 14:09:37 avagent Error <6210>: Unable to register '/clients/VDPApps\sqlvm' with MCS at 192.x.x.x:28001

2017-05-19 14:09:37 avagent Error <6163>: Registration error: MCS returned error code 110, '* 110 * Re-registration of account /clients/VDPApps/sqlvm.happycow.local is not permitted'

2017-05-19 14:09:37 avagent Error <7531>: Unable to register /clients/VDPApps/sqlvm with Administrator 192.x.x.x:28001

So, to deactivate this client, from the SSH of the VDP run the below command:

# mccli client edit --activated=false --name=/clients/VDPApps/<client-fqdn> 

Then retry the install of the agent and it should complete successfully.

Hope this helps.

Shell Command: Understanding tee

By default when you run a command, the output is reflected onto stdout. Consider this simple example,

# echo "Hello World"

The output would be:
Hello World

The output is displayed on stdout which is my terminal.

In some cases, we will have a need to run the command, view the output and simultaneously save the output to a log file. This is where tee comes in. Consider tee as a fork in a road. The output goes both ways, one to stdout and the other to a file

Syntax:

# some command | tee <file_1> <file_2>.....<file_n>

The output of some command is displayed on stdout and stored in one or more files.

tee comes with a couple of switches. The most simple tee command is:

# echo "Hello World" | tee file.txt

The output is Hello World on stdout and if you cat file.txt you will see the output Hello World
If you run the same command with a different echo text to the same file, then the existing content is replaced.

So if you want to save the existing content and append new output, then use the -a switch

# echo "Hello World" | tee file.txt

# echo "Today is a good day" | tee -a file.txt

When you run the first command, the output on stdout is Hello World and the same output is seen when you cat file.txt

When you run the second command, the output on stdout is Today is a good day and the output when you cat the file.txt is:
Hello World
Today is a good day

Next is tee with -i or --ignore-interrupts which ignores the SIGINT which would be your Ctrl+C
Sigint sends an interrupt signal to terminate a foreground process, generally when you press Ctrl+C, but you want tee to be terminated gracefully. In short, -i ignores Ctrl+C SIGINT