Tuesday, 18 April 2017

VDP - Avmar Migration: Part-4: Migrating Data And Jobs From VDP To Avamar

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

In this part, we will replicate VDP backup data to Avamar and then migrate the jobs. First, to replicate the data from VDP to Avamar, we create a regular replication job. Let's have a look at this.

Connect to the VDP appliance from the web client and click the Restore tab. To give you a brief overview, here is the restore points for my VM-A


Click the Replication tab and click Replication job actions. and click Create New Replication Job.


You will be presented with the below wizard. I want to replicate my current Image backups, so the first option and click Next.


I will replicate the individual client, and select VM-A and then click Next.


Select which backup schedule you would like to replicate. This is as per your need and then click Next.


Enter the AVE FQDN/IP as the hostname, the port for replication is 29000. The username to be used is repluser and the password for repluser is the one you setup during the Avamar configuration. Click Verify Authentication. Once the connection test is completed successfully, click Next.


Select how frequently you need to replicate. This does not really matter here, because it is going to be one time replication, because post the migration source VDP will be in a unusable state. Click Next.


Select a retention policy for the replicated backup as per your requirement and click Next.


Provide a name to this replication job and then click Next and then Finish the wizard.


Once the job is created, select the job and click Replicate Now to begin the task.


Once the replication is completed, you can see the replicated data in the Avamar end.


You will still notice there is a red X for Backup Jobs as the jobs are not transferred. It is only the VDP backup data that is replicated to Avamar.


Next we will begin migrating the jobs. To do this, open a browser and go to:
https://vdp-ip/fqdn:8543/vdp-configure

Once on this page, click Avamar Migration and click Migrate.


Check the Terms and conditions to proceed further and click Next.


Provide the AVE FQDN, the root credentials would be for Avamar root and not the OS root password. Click Verify Authentication.


Once the authentication succeeds, click OK and then click Next.


You can select the backup jobs you need to migrate here. 


You can switch the tab to Backup Verification and select the required verification jobs and then click Next.


A validation will be performed and if everything is successful, click Next.


You will be prompted for migration. Click Yes to begin the migration.


You can then monitor the migration progress.


Then you can review the migration status. Click Close.


Now the status is, Jobs have been migrated and the appliance is disabled.


The same is seen in the VDP page in web client as well.


On the Avamar end, you can see the jobs have now been migrated successfully.



So, in the end, the VDP would be used only for performing restore operations with the existing data. And once new valid backups are performed by Avamar, you can discard VDP completely.

Hope this series was helpful. Let me know your thoughts in the comments.

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2

Previously we configured AVE services. Next we need to download the Avamar Client, install it and then configure a vCenter and deploy a proxy. Let's look at this step by step.

Let's start off by downloading the Avamar Client. To get this, go to the below URL:
http://avamar-fqdn

You should see the below screen:


Click Downloads and you should see the below screen. First download the jre.exe and have it installed on the machine where you plan to run the Avamar Client. Then download AvamarConsoleMultipleWindows.exe

Run the installer, it's a pretty straightforward wizard to follow.


Once the client installation is done, open the Avamar client and you will be presented with the below login screen. The Server would be FQDN or IP of your AVE. The Username would be MCUser and the Password for the same with / Domain.


Once the login is successful, we will see the below Avamar management UI.


Notice there are no backup jobs, hence a red X for Backup Jobs tab. Now, we will have to configure vCenter to the AVE. Click Administration from the management UI and you will see a new window:


The vCenter client, should always be added to the / domain. So select the ave and then click Actions > Account Management and then click New Client.


You will see the following section. Select the Client Type as VMware vCenter. Provide the vCenter FQDN or IP for Client name. For username provide an admin user defined on the vCenter level. Disable vCenter Certificate parameter before performing this. You can find this step in this link here.
Click OK and the vCenter should be registered successfully to the AVE machine.


You will notice the vcenter-prod (my vCenter) under the ave domain added as a client.


Next, you will have to deploy a proxy, without this your migration pre-check will fail. In the same window, click VMware and Proxy Deployment Manager.


Here you can input your required setting and create a recommendation.


Click Apply and then click New Proxy and you will be presented with the below window. Enter a name for the proxy. The domain would be your /vcenter-fqdn/VirtualMachines. Provide an IP for the proxy, datastore and a network. Please note the proxy should be resolvable before performing the deployment. Click Save.


The proxy deployment will begin and you can monitor the progress here as well as the vCenter Recent Tasks pane.


This should be it. Until now, you have deployed AVE ovf, configured it, downloaded the client and configured it to a vCenter and added a proxy.

Next we will look at the migration of data and jobs from VDP to AVE.

VDP - Avmar Migration: Part-4: Migrating Data And Jobs From VDP To Avamar

Monday, 17 April 2017

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2

VDP - Avmar Migration: Part-1: Deploying Avamar Virtual Edition 7.2

In part 1 we saw how to deploy and setup the basic Avamar 7.2 machine. In this article, we will have the appliance configured for usage.

Open a browser and type in:
https://avamar-server-ip/fqdn:7543/avi/avigui.html

This should bring up the below screen and you can login with root as the user and changeme as the password.


Post the login you will see the below page. Here on the top left corner you will see a lock icon. Click this and enter the password as Supp0rtHarV1.  Once unlocked, click Install to begin the package installation.


The initialization will take a few minutes to complete and once done you will be presented with the below window. Here, you will need to fill all the fields with the red exclamation marks.

Under Server Settings, the Avamar Server Address should be the hostname and select an appropriate time zone.

The number of storage node I will be using is 0, just like how my VDP is setup. A single node server. This is the node you see when you run status.dpn


Fill in the remaining fields and click Continue on the bottom right. This should begin the package installation like seen below:


Once the configuration is complete you will see the below task. At this point, your AVE setup is complete.

Now you can SSH into the AVE machine with admin credentials and run the below command to verify system and service status.

admin@ave:~/>: dpnctl status
Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: Maintenance windows scheduler status: suspended.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.

admin@ave:~/>: status.dpn
Tue Apr 18 00:49:01 IST 2017  [AVE] Mon Apr 17 19:19:01 2017 UTC (Initialized Mon Apr 17 17:51:04 2017 UTC)
Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk
0.0   10.109.10.169   7.2.1-32  ONLINE fullaccess mhpu+0hpu+0hpu   1 false   0.49 5209    33104   0.0%   0%(onl:8  )  0%(onl:8  )  0%(onl:8  )
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1492451464@00:50:56:9A:52:F6

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full

Last checkpoint: cp.20170417180550 finished Mon Apr 17 23:36:09 2017 after 00m 19s (OK)
No GC yet
No hfscheck yet

Maintenance windows scheduler capacity profile is active.
  WARNING: Scheduler is WAITING TO START until Wed Apr 19 20:30:00 2017 IST.
  Next backup window start time: Thu Apr 20 20:00:00 2017 IST
  Next maintenance window start time: Thu Apr 20 08:00:00 2017 IST

That should be it with the configuration part. Next, we will look into how to Configure AVE to vCenter Server. 

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

VDP - Avmar Migration: Part-1: Deploying Avamar Virtual Edition 7.2

Since VMware announced, the end of vSphere Data Protection, there is a choice to migrate existing deployment to EMC Avamar. More about EOA can be found in this link here.

In this article, we will be looking at deploying Avamar Virtual Edition 7.2. You can go ahead and download the required version of AVE from EMC download portal. The version I will be using is Avamar 7.2.1.

Login to Web Client or vSphere Client, Select the ESXi host where you want to deploy your AVE and select File > Deploy OVF. Browse the location for the AVE download and add the file. Click Next.


Review the details of the OVF template and click Next.


Accept the EULA and click Next.


Provide a name for this AVE virtual machine. Click Next.


If available and required select a resource pool in which you want to place this VM. Click Next.


Select the datastore where you want to deploy this. Remember the vmdk bundled with AVE is just the OVF, the data drives are configured later just like a VDP appliance. Click Next.


Select a disk provisioning type. Thick provision is recommended. Click Next.


Select a network where this AVE should be connected to. Click Next.


Review the changes and click Finish. Do not check Power On after deployment, because there are couple of steps to be done once the OVF deployment is completed.


Just like VDP, the AVE comes with 4 supported available backup storage. You can refer the below table to size your AVE accordingly.

Once you choose the deployment type, you will need to refer the below table to plan the drive sizes. Just like in VDP a 512GB deployment will have 3 drives of 256 GB each. The additional space is for the checkpoint maintenance overhead.

So the rule goes like:
Total size = GSAN capacity + 1/2 of GSAN capacity.

GSAN capacity would be the actual space for storing backup data.


So go ahead and add three disks manually (Depending on your AVE configuration) to this VM. Only Thick Provisioning is supported for AVE. I will be using Thin because of space constraints.


Once the drives are added, power on the AVE virtual machine.

The default login is root and changeme

First, we will have to configure network settings for the AVE machine. Post the login to AVE from VM console, run yast2 to begin the networking configuration. You will see a similar interface:


Select Network Devices and then Network Services to begin the network configuration wizard and you should see something similar:


You will need to Set IP Configuration in Overview, Hostname / DNS settings and Gateway under Routing.

Once the appliance is configured with network, restart the guest and then verify the network by ping and nslookup. If this works good proceed to Part 2 in the below link.

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2

Friday, 14 April 2017

VDP Configure Page Reports - Server Is Still Starting

You might sometimes restart your appliance and you will be presented with the message:

The server is still starting. Depending on the configuration, this could take up to 25 minutes. Try again later.

No matter, how many times you try to login you will run into the same message.


Again, if you look at the vdr-configure.log, you will notice the following:

2017-04-15 05:44:49,242 INFO  [http-nio-8543-exec-3]-services.LoginService: Login service called with action: [login]
2017-04-15 05:44:49,243 INFO  [http-nio-8543-exec-3]-services.LoginService: Checking if the server is in a running state...
2017-04-15 05:44:49,243 INFO  [http-nio-8543-exec-3]-services.LoginService: Server is not running
2017-04-15 05:45:06,592 WARN  [pool-21-thread-1]-backupagent.BackupAgentUpdaterImpl: No proxy-clients are available.

This does not really help much to understand that what is going on.The cause here is due to missing .av_sys_state_marker_running file. I guess this file records the state of the VDP appliance. If this file goes missing, the server is unable to determine the state, which is why vdr throws up "Server is not running" in the logs. 

The file is located under /usr/local/avamar/var

Go to this directory and recreate this file using:
# touch .av_sys_state_marker_running

Post this, refresh the vdp-configure page and you should have access.

Failed To Start Internal Proxy In VDP 6.x

Mostly after an upgrade most of your backups fail with a status of "No eligible proxies" or "No data"
You will not be able to run on demand backups in some cases and this would fail with an error "Adhoc Backup Request Error - Exception"

root@vdp-dest:/data01/home/admin/#: mccli client backup-dataset --domain=/vcenter-prod.happycow.local/VirtualMachines --name=VM-C
1,22253,Client Adhoc Backup Request Error - Exception.

If you try to enable Internal proxy from the vdp-configure page, it will fail with the below error:


In the vdr-configure.log you will notice the following:

2017-04-15 03:50:52,463 ERROR [pool-22-thread-1]-cmdline.RuntimeExecImpl: avagent Info <5008>: Logging to /usr/local/avamarclient/var/avagent.log
2017-04-15 03:50:52,463 ERROR [pool-22-thread-1]-cmdline.RuntimeExecImpl: avagent Error <7531>: Unable to register clients/vdp-dest with Administrator 127.0.0.1:28001
2017-04-15 03:50:52,464 ERROR [pool-22-thread-1]-cmdline.RuntimeExecImpl:  'Could not reconcile proxy with vCenter.' (203)
2017-04-15 03:50:52,464 ERROR [pool-22-thread-1]-cmdline.RuntimeExecImpl: avagent Info <5008>: Logging to /usr/local/avamarclient/var/avagent.log

You will see vCenter connections down if you run the below command:
# mccli server show-services

You will something similar to:

0,23000,CLI command completed successfully.
Name                               Status
---------------------------------- -----------------------------
Hostname                           vdp-dest.happycow.local
IP Address                         10.109.10.167
Load Average                       0.24
Last Administrator Datastore Flush 2017-04-15 04:45:00 IST
PostgreSQL database                Running
Web Services                       Error
Web Restore Disk Space Available   256,417,868K
Login Manager                      Running
snmp sub-agent                     Disabled
ConnectEMC                         Disabled
snmp daemon                        Disabled
ssh daemon                         Running
Data Domain SNMP Manager           Not Running
Remote Backup Manager Service      Running
RabbitMQ                           Not Running
Replication cron job               Not Running
/vcenter-prod.happycow.local       5 vCenter connection(s) down.

If you try to register proxy from the command line using the below command, it will fail as well. 
# /usr/local/avamarclient/etc/initproxy.sh start

avagent.d Info: Stopping Avamar Client Agent (avagent-vmware)...
avagent.d Info: Client Agent stopped.
avagent Info <5008>: Logging to /usr/local/avamarclient/var/avagent.log
avagent Error <7531>: Unable to register clients/vdp-dest with Administrator 127.0.0.1:28001
 'Could not reconcile proxy with vCenter.' (203)
avagent.d Info: Client activation error.
avagent Info <5008>: Logging to /usr/local/avamarclient/var/avagent.log
avagent Info <5417>: daemonized as process id 351
avagent.d Info: Client Agent started.

Registration Failed.
initproxy.sh FAIL: registerproxy failed

The cause:
This is because, there is a key called as "ignore_vc_cert" which will be flipped to false. The VDP will always be waiting for process to acknowledge the certificate warning which will never work and hence the proxy fails to start.

The fix:
1. Run the below command to verify the key value:
# grep -i ignore /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

The output should be similar to:
     <entry key="ddr_ignore_snmp_errors" value="false" />
     <entry key="email_logs_tar_cmd" value="tar -cz --atime-preserve=system --dereference -- ignore-failed-read --one-file-system --absolute-names" />
      <entry key="ignore_vc_cert" value="false" />

2. Edit this mcserver.xml file and replace the ignore_vc_cert value to true and save the file

3. Switch to admin mode of VDP (sudo su - admin) and restart the mcs using:
# mcserver.sh --restart

4. Register the internal proxy from GUI and it should work successfully and none of the vCenter connections will be reported as down.

Hope this helps.

Tuesday, 11 April 2017

Unable To Configure VDP To vCenter - Unable to find this VDP in the vCenter inventory

So, you might run into issues where you are unable to configure VDP to vCenter and you run into this error.
Unable to find this VDP in the vCenter inventory



In the vdr-configure.log you will notice the following. Again, for all issues with vdp-configure page refer the vdr-configure.log

2017-04-10 10:41:13,365 WARN  [http-nio-8543-exec-2]-vi.VCenterServiceImpl: No VCenter found in MC root domain
2017-04-10 10:41:13,365 INFO  [http-nio-8543-exec-2]-reconfig.VcenterConfigurationImpl: Failed to locate vCenter Client in Avamar, reconfiguration is required
2017-04-10 10:41:13,365 INFO  [http-nio-8543-exec-2]-sso.VmwareSsoServiceImpl: Getting SSL certificates for https://psc-prod:7444/lookupservice/sdk
2017-04-10 10:41:13,715 INFO  [http-nio-8543-exec-2]-services.VcenterConnectionTestService: Finished vCenter Connection test with result:
                <?xml version="1.0"?><vCenter><certValid>true</certValid><connection>true</connection><userAuthorized>true</userAuthorized><ave_in_vcenter>false</ave_in_vcenter><switch_needed>true<
/switch_needed><persistent_mode>true</persistent_mode><ssoValid>true</ssoValid><httpPortValid>true</httpPortValid></vCenter>

2017-04-10 10:41:13,025 WARN  [http-nio-8543-exec-2]-vi.VCenterServiceImpl: Failed to get root domain from MC
2017-04-10 10:41:13,025 WARN  [http-nio-8543-exec-2]-vi.VCenterServiceImpl: No VCenter found in MC root domain
2017-04-10 10:41:13,025 INFO  [http-nio-8543-exec-2]-vi.ViJavaServiceInstanceProviderImpl: visdkUrl = https://vc-prod:443/sdk
2017-04-10 10:41:13,337 INFO  [http-nio-8543-exec-2]-util.UserValidationUtil: vCenter user has sufficient privileges to run VDP.
2017-04-10 10:41:13,339 INFO  [http-nio-8543-exec-2]-network.NetworkInfoApi: Found IP Address: [10.116.189.178] link local? [false], site local? [true], loopback? [false]
2017-04-10 10:41:13,339 INFO  [http-nio-8543-exec-2]-network.NetworkInfoApi: Found IP Address: 10.116.189.178

2017-04-10 10:41:13,353 ERROR [http-nio-8543-exec-2]-vi.ViJavaAccess: getPoweredOnVmByIpAddr(): Cannot determine appropriate powered on AVE virtual machine with IP Address [10.x.x.x] since there exist many of them (2): type=VirtualMachine name=vdp-vm mor-id=vm-208, type=VirtualMachine name=Windows-Jump mor-id=vm-148


So in this case, 10.x.x.x is the IP of my VDP machine and there is a duplicate IP used by another VM in the vCenter and this is Windows-Jump. If this is the case, determine if you can remove the duplicate IP or change the IP of the VDP appliance. The configuration test should then complete without issues. 

Hope this helps.