Backup VDP Data To An External Drive

Well, you have got your new vSphere Data Protection set up and have kicked off a few successful backups. The backup data is stored in your VDP datastore safely and you can restore it any time you want to. Let's say that your VMFS volume hosting your VDP data drives went corrupt, this means that all your precious backed up data is lost. So most of the enterprises look for means to move those backup disk of VDP to an external drive.

Now, there are two ways of achieving this. 

1. Using a script to make a copy of your data disks of VDP to a tape drive or your local machine. The disadvantage of this is that you will have to power off your backup appliance in order for the disks to be copied over. So, if you have one VDP appliance in your environment, your backup is down until the copy completes. 

2. Deploy another VDP appliance and configure replication between the primary and the secondary appliance. The primary is the one that performs your scheduled backups and then replicates this data to the secondary appliance. So, daily backups occur on the primary machine. Replication occurs on the weekends to the secondary appliance. The script runs on the secondary appliance for the rest of the week, as the secondary is not receiving any replication traffic and neither it is running any backup jobs, it can afford a downtime. 

The script:

#Connects to the vCenter Server hosting the backup appliance

Connect-VIServer -Server <vCenter_IP> -User administrator -Password <Password>

#Shuts down the VDP appliance

Shutdown-VMGuest -VM "<VDP_Appliance_Name>" -Confirm:$false

#Sleep time after which the copy script kicks in. Value is in seconds

Start-Sleep -s 600

#Copy script

Copy-DataStoreItem vmstore:\<Data_centername>\<VDP_Datastore_Name>\"<VDP_Appliance_Name> 6.1"\*.vmdk D:\VDP\vdp01 -Force

#Starts the appliance after the copy is done

Start-VM -VM "vSphere Data Protection 6.1" -Confirm:$false

Here, I am copying the VDP vmdk only files to D drive on my local computer. If you want to copy the remaining files, then simply replicate the #Copy script lines and change *.vmdk to the other file extensions (*.vmx, *.nvram etc)

Sample output during the Power Off:

Sample output during the copy:

Once the copy completes, you can verify the contents of the VM in the local drive/tape connected to your machine. 

**Try at your own risk if you are running a production environment. All the tests for the script was done in my lab**

Cannot Connect VDP To Web Client. SSO Server Could Not Be Found

You have setup a new VDP appliance in your environment or have an existing one. And suddenly you run into issues regarding connecting this appliance to the Web Client. Every time you choose the appliance from the drop-down and hit connect, you run into the error:

"Could not connect to the requested VDP appliance. The SSO Server could not be found. Please make sure the SSO and DNS servers are available on the network and all settings are configured properly"

Pre-checks and solutions:

1. DNS should be working good. Check if you are able to resolve the IP and FQDN of the VDP appliance along with your vCenter server from the machine where you are trying to connect. If DNS resolution is running into issues, then have the DNS issue fixed, and then try the reconnect. 

2. Time on the VDP appliance to ESXi host to your SSO server must be synced. If time is not synced then your will run into NTP error message during the connect. However, please make sure that the time is synced across all the three components. Use the command "date" in VDP to list the current time. 

3. You can try adding the VDP appliance and vCenter IP/FQDN entry in the /etc/hosts file of the VDP appliance and then try a reconnect. 

For me, all of the above pointers were working good, and yet I was unable to connect the appliance to the Web Client. 

Restarted the tomcat service on the VDP appliance and I was able to successfully connect the appliance to the web client. To restart this:

#: emwebapp.sh --restart

That's pretty much it!

You can refer this blog here for more information and this KB here for further troubleshooting if these steps do not help.

VDP CLI Commands

Here is a set of commands for various operations that you can perform from the SSH of the VDP appliance. I will update this article as I come across new commands.

dpnctl status all

//Shows output of all the services of the VDP appliance.

//Sample output:

dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: axionfs status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

status.dpn

//Shows appliance access, status, last checkpoint, last hfs check and deduplication space reports.

//Sample output

Thu Apr  7 20:02:17 IST 2016  [vdp.vcloud.local] Thu Apr  7 14:32:17 2016 UTC (Initialized Mon Apr  4 18:47:03 2016 UTC)
Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk
0.0   192.168.1.206  7.2.80-94  ONLINE fullaccess mhpu+0hpu+0hpu   1 false   0.38 3420   102774   0.6%   0%(onl:13 )  0%(onl:12 )  0%(onl:12 )
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1459795623@00:50:56:B9:6B:3C

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full

Last checkpoint: cp.20160407033314 finished Thu Apr  7 09:03:33 2016 after 00m 19s (OK)
Last GC: finished Thu Apr  7 08:00:39 2016 after 00m 16s >> recovered 64.05 MB (OK)
Last hfscheck: finished Thu Apr  7 09:03:05 2016 after 02m 01s >> checked 32 of 32 stripes (OK)

Maintenance windows scheduler capacity profile is active.
  The backup window is currently running.
  Next backup window start time: Fri Apr  8 20:00:00 2016 IST
  Next maintenance window start time: Fri Apr  8 08:00:00 2016 IST

capacity.sh

//Shows change of data for the list of backed up VMs and a list of top 3 high change VMs in backup jobs.

//Sample output

Date          New Data #BU       Removed #GC    Net Change
----------  ---------- -----  ---------- -----  ----------
2016-04-04     1047 mb 1            0 mb           1047 mb
2016-04-05     3311 mb 4            0 mb 1         3311 mb
----------  ---------- -----  ---------- -----  ----------
Average        2179 mb              0 mb           2179 mb

Top 3 High Change Clients:
--------------------------
Total for all clients                     4358 mb      100.0%
  Windows_2008_UDmaxWmHw0pC0PB5bXgAKg       3306 mb       75.8%
  CentOS7_UDlmW0iZnAAWleQvAdTtog          1052 mb       24.2%

mccli activity show

//Shows status of running backup jobs.

//Sample output

ID               Status  Error Code Start Time           Elapsed     End Time             Type             Progress Bytes New Bytes Client       Domain
---------------- ------- ---------- -------------------- ----------- -------------------- ---------------- -------------- --------- ------------ ----------------------------
9146003975305509 Running 0          2016-04-07 20:05 IST 00h:00m:16s 2016-04-08 20:05 IST On-Demand Backup 0 bytes        0%        Windows 2008 /192.168.1.1/VirtualMachines

mccli activity get-log --id=XXXXXXXXXX

//Shows the logs for the backup job for the backup ID. Backup ID can be found from the mccli activity show

mccli activity cancel --id=XXXXXXXXXX

//To cancel the actively running backup job.

//Sample output

0,22205,Backup cancelled via console
Attribute   Value
----------- ----------------
activity-id 9146003975305509

mccli client show --recursive=true

//Show the registered clients with the VDP.

//Sample output
Client           Domain                       Client Type
---------------- ---------------------------- ------------------------------------
Windows 2008     /192.168.1.1/VirtualMachines Virtual Machine
192.168.1.1      /192.168.1.1                 vCenter
vdp.vcloud.local /clients                     VMware Image Proxy with Guest Backup

mccli backup show --name=/vCenter/VirtualMachines/VirtualMachine_Name --recursive=true

//Shows the successful backup and the restore point created for the same.

//Sample Command

 mccli backup show --name=/192.168.1.1/VirtualMachines/CentOS7 --recursive=true

//Sample Output
Created                 LabelNum Size    Retention Hostname         Location
----------------------- -------- ------- --------- ---------------- --------
2016-04-07 20:31:34 IST 1        16.0 GB DWMY      vdp.vcloud.local Local

mccli client backup-dataset --name=VM_NAME --domain=/VM/DOMAIN

//To start backup of a VM from command line.

//Sample Command:

mccli client backup-dataset --name="Windows 2008" --domain=/192.168.1.1/VirtualMachines

//Sample output

Attribute   Value
----------- -----------------------------------------
client      /192.168.1.1/VirtualMachines/Windows 2008
activity-id 9146004043408409
dataset     /VMware Image Dataset

cplist

//Shows the checkpoint for the appliance. Valid and Not validated checkpoint

//Sample output

cp.20160407033040 Thu Apr  7 09:00:40 2016   valid rol ---  nodes   1/1 stripes     37
cp.20160407033314 Thu Apr  7 09:03:14 2016   valid --- ---  nodes   1/1 stripes     37

mccli server show-prop

//Shows VDP details. Useful if you cannot connect to VDP from the web client

//Sample output

Attribute                        Value
-------------------------------- ----------------------------
State                            Full Access
Active sessions                  1
Total capacity                   575.9 GB
Capacity used                    5.3 GB
Server utilization               0.9%
Bytes protected                  0 bytes
Bytes protected quota            Not configured
License expiration               Never
Time since Server initialization 2 days 20h:04m
Last checkpoint                  2016-04-07 09:03:14 IST
Last validated checkpoint        2016-04-07 09:00:40 IST
System Name                      vdp.vcloud.local

mccli server show-services

//Another command to check service status

//Sample output

Name                               Status
---------------------------------- ---------------------------
Hostname                           vdp.vcloud.local
IP Address                         192.168.1.206
Load Average                       0.31
Last Administrator Datastore Flush 2016-04-07 19:45:00 IST
PostgreSQL database                Running
Web Services                       Error
Web Restore Disk Space Available   266,182,888K
Login Manager                      Running
snmp sub-agent                     Disabled
ConnectEMC                         Disabled
snmp daemon                        Disabled
ssh daemon                         Running
Data Domain SNMP Manager           Not Running
Remote Backup Manager Service      Running
RabbitMQ                           Not Running
Replication cron job               Not Running
/192.168.1.1                       All vCenter connections OK.

emwebapp.sh --restart / --stop / --start

//Command to restart tomcat service for VDP. If this service is stopped you cannot connect VDP to Web Client or access VDP GUI page on 8543.

//Sample successful start

INFO: Copying CST libs to tomcat
Administrator Server is not running.
Starting the database server.
Waiting for postmaster to start ...Started
No script specified

Database server is still running...
Starting tomcat
/usr/local/avamar-tomcat/bin/startup.sh
Using CATALINA_BASE:   /usr/local/avamar-tomcat
Using CATALINA_HOME:   /usr/local/avamar-tomcat
Using CATALINA_TMPDIR: /usr/local/avamar-tomcat/temp
Using JRE_HOME:        /usr/java/latest
Using CLASSPATH:       /usr/local/avamar-tomcat/bin/bootstrap.jar:/usr/local/avamar-tomcat/bin/tomcat-juli.jar
Using CATALINA_PID:    /usr/local/avamar-tomcat/avamar-tomcat.pid
Tomcat started.

dpnctl start gsan / mcs / axionfs

//Start each of the services individually

avmaint hfscheck --ava --full

//To run a full integrity check on your VDP appliance

//Sample successful output 

<hfscheck
  checkpoint="cp.20160407033314"
  status="waitcgsan"
  type="full"
  checks="full"
  elapsed-time="70"
  start-time="1460064369"
  end-time="0"
  check-start-time="0"
  check-end-time="0"
  generation-time="1460064439"
  percent-complete="0.00">
  <hfscheckerrors/>
</hfscheck>

avmaint hfscheckstatus --ava

//To check hfscheck status. This tells the last cp that was created during the integrity check and the status.

//Sample output

checkpoint="cp.20160407033314"
status="completed"

Re-register VDP 6.1 to vCenter Server

Sometimes, there might be a need to re-register your VDP appliance to your vCenter server, maybe to use a different user account for registration or some issues with the vCenter. The registration process is quite easy and will not affect any of your backup jobs or the backup data present in your deduplication store. 

To Re-register your VDP appliance to vCenter, follow the below steps:

From the below screenshot you can see the backup job that is already present on my appliance prior to the re-registration. 

Next, you need to go to your vdp-configure page, which is available at the below link.

https://VDP_IP/vdp-configure

Login to your appliance with your root credentials and you will come across the below page. Click the gear icon and select vCenter registration. 

Please read the below message. Do not make any changes to vCenter with regards to vCenter host-name, IP, port number. This will cause your backup jobs to be lost. 

However, re-registering with a different user should not cause any issues. 

Provide the new user details and keep your vCenter details the same. Click Next, review the changes and click Finish. 

The below task will be started during the re-registration process. Once the task is done, it will reconnect to web client and log you out of your vdp-configure session. 

Now, login back to your web client, go to vSphere Data Protection. Connect the required appliance to web client and go to the backup tab and you will notice your backup job is still retained. 

That's it!

Cannot Open vdp-configure Page Or Check Status of VDP Services

So when you try to open the :8543/vdp-configure page you will receive the message:

"This site can't be reached. ERR_CONNECTION_REFUSED"

When you open SSH to this VDP appliance and check the status of the services, this will also throw up an error. Run the below command to check the VDP service status:

root@vdp:~/#: dpnctl status

The error you will receive is:

mkdtemp: private socket dir: No space left on device.

I tried to run the command to start the webservices, which is:

root@vdp:~/#: emwebapp.sh --start

Which also failed with an error:

Waiting for postmaster to start ...........Failed to connect DBI:Pg:dbname=postgres;port=5558.ERROR: Failed to start the database.

Interesting!
I ran the below command to check the space on the VDP appliance partition.

root@vdp:~/#: df -h

Here the output I noticed the partition, /dev/sda2 was at 100 percent used.

Run the below command to list directories within each other with largest used space:

root@vdp:~/#: du -h --max-depth=1 <directory>

Upon performing this, I found the below directory occupying nearly 40 percent of space on sda2

root@vdp:~/#: /usr/local/avamar-tomcat-7.0.42/logs

Removed all the old log files from this directory.

Also, if the space does not change even when the logs from the above directory is removed, then you need to check the following directory:

root@vdp:/usr/local/avamar-tomcat-<your_version>/webapps/ROOT

You will see a logbundle.zip file which is a manually generated log file. You can go ahead and remove this log.zip file. Do not remove any other file in this directory.

Prior to removing the logs, stop the VDP services using the below command:

root@vdp:~/#: dpnctl stop

However, this command also failed due to unavailable space. If this occurs, go ahead and remove the files without stopping the service. I risked this, however, the log files cleared out and space was freed and I was able to start the web services for VDP and login to the GUI of the appliance.

Cheers!

Unable To Register VDP to vCenter in the vdp-configure Page

After deploying the OVF template for a new VDP appliance, we will have to go to the vdp-configure page to get the appliance configured to the vCenter.

Here, in the vCenter registration page, after entering the username, vCenter details and try to test connection, you run into the below error.

"Unable to verify vCenter listening on specified HTTP Port. Please re-check values and try again"

So, here I was trying to configure VDP appliance with a port number of 80 for http and 443 for https.

However, the vCenter is running on a custom port of 81 and 444.

You can login to your vCenter, 6.0, Select the Administration tab > vCenter Server Settings > Advanced Settings.

Here there are two parameters which talks about your vCenter ports. They are:

config.vpxd.rhttpproxy.httpsport 443
config.vpxd.rhttpproxy.httpport 80

443 and 80 are the default ports. If they are different, then we are using custom ports and we need top open this port on the VDP appliance firewall.

You can use telnet to check the connection between the appliance and vCenter.
Run the command from the SSH of the appliance

telnet <server_IP> <port_number>

To perform this:
1. Open a SSH to the VDP appliance.
2. Change your directory to:

#: cd /etc/

3. Open the file "firewall.base" in a vi editor
4. Locate the line:

exec_rule -A OUTPUT -p tcp -m multiport --dport 53,80

5. Add your custom http and https port value here and save the file.
6. Restart the firewall service using the following commands:

#: service avfirewall stop

#: service avfirewall start

Register the appliance again and make sure you give the custom ports in the http and https field during configuration.

That's it!

Migrate Networking From Distributed Switch To Standard Switch

Written by Suhas Savkoor

In the previous article here, we saw how to migrate ESXi networking from Standard Switch to Distributed Switch. In this one, we will perform the reverse of this.

Step 1:
This is the setup that I have for my vDS after I had it migrated.

Here I have 2 portgroups, one for my virtual machine and one for my vmk management port-group. And both of these are connected to two uplinks, vmnic0 and vmnic1

Step 2:
Before creating a standard switch, I will be removing one of the vmnic (Physical Adapter) from the vDS as I do not have any free uplinks to add to the standard switch. Select Manage Physical Adapters and Remove the required uplink.

Step 3:
Now let's go ahead and create a new Standard Switch. Select the vSphere Standard Switch and click Add Networking

Step 4:
Choose Virtual Machine as the port-group type.

Step 5:
Select the available uplink that needs to be connected to this standard switch and click Next

Step 6:
Provide a Network Label to the virtual machine port-group on the standard switch.

Review the settings and complete the create and now you will have one Standard Switch with one virtual machine port-group connected to an uplink. It's now time to begin the migration.

Step 7:
Go back to distributed switches section and select Manage Virtual Adapters

Step 8:
Select the required vmk and click Migrate

Step 9:
Select the required vSwitch as to where you want to migrate this port-group to.

Step 10:
Provide a Network label for this vmk port-group on the standard switch. If you are using any VLAN for the vDS port-group for this vmk, specify the same in the VLAN section to replicate this on the standard switch. Else the migration fails.

Review and complete and you have the management vmk migrated off the distributed switch to the standard switch.

Step 11:
To migrate virtual machine's networking, go to Home > Networking > Right click the vDS and select Migrate Virtual Machine Networking

Step 12:
The source would be the VM port-group on the vDS in my case is, dvPortGroup ad the destination is the standard switch port-group which we created recently, VSS VM Portgroup

Step 13:
Select the virtual machines that you want to migrate.

Review and finish and once the migrate completes, you can now check the standard switch configuration to verify everything is migrated successfully.

Well, that's it!