VDP File Level Recovery Client

So, you would be familiar with restoring virtual machines with VDP which is done from the Restore tab of the VDP appliance from the Web Client GUI. However, there are cases, where the necessity is to restore only certain files or folders and not the entire virtual machine or virtual disk. In this case, instead of using disk restore or virtual machine restore we perform File Level restore. 

Like your VDP configure page, FLR also has a client. The FLR client has to be accessed on the virtual machine where you want to perform the file restores. The URL for this would be:

https://VDP-IP:8543/flr

The login screen would look something like this:

This is the simple login page and there is an option for advanced login as well. We will have a look at the advanced login a little later in this article. 

For the simple login, you will have to provide the Windows machine's local administrator credentials. Now, for a small demo, on the machine where I have accessed the FLR, I had a file under "Desktop" directory called "Critical data" which had some data called "Critical Text". To give a little more background, once this file was created, a regular backup was executed for this virtual machine. The backup completed successfully. However, some hours later the file was lost. Now, we will be restoring that file using the FLR client. 

So I have logged in with my local admin credentials to the FLR client and this is the first screen I would see after a successful login. 

I have just one restore point created for this virtual machine since only one backup was taken. If you have multiple restore points after a set of backups, you can choose the restore point that you think is the best and select the option called Mount. 

You will then see this directory view. I will expand the drive and folders until, the necessary file is located. So my file was created under C > Users > Administrator > Desktop. Here the critical data text file is seen.

I will select only this file to restore and select the "Restore selected files" option. Upon selecting that you will have to provide a location as to where you want to restore the file. I will select the destination where it was originally residing. If you want to specify a different destination, you can do the same as well here. Click the Restore option once the directory is decided. 

You will receive a generic prompt stating that the operation might be time consuming. Select Yes to start the restore. 

Click Monitor Resources tab to check the restore status. Here the progress is not shown, and the only view available is as the one seen below:

From the vSphere client you can monitor the actual status of the restore.

Once the restore is completed successfully you can go to that directory and verify the file is restored successfully. 

The next option is restoring using the advanced login method. The advanced login method is used when, the machine where you actually want to restore the file has no access. In this case, with the advanced login, you will provide the credentials of the vCenter along with local credentials of the machine. 

The login would something like below:

Here the restore point screen looks a bit different. Instead of seeing only the restore point of the virtual machine, you will be seeing all the restore points for the virtual machines which were backed up.

Click the drop down for the virtual machine required, or you can also filter by name to locate it quickly. Click the required restore point from the list after selecting the drop-down and click mount. 

The remaining process is going to be same, you can restore the file to the same location or you can restore the file to a different location. 

One more thing to note:

Let's say, initially you had a text file with contents ABC. Then you took a backup of the virtual machine. After the backup, you made changes to the text file so your contents are now ABCDE. Now, if you restore this file on the same directory where the original file exists then the new contents of the file will be over written with the contents present in the restored file. So make sure that restore is done appropriately. 

That's it!

To know more about limitations of FLR you can click this link here to read the VDP admin guide. (See page 152)

Error While Binding iSCSI To VMKernel Adapters: IscsiManager.QueryBoundVnics

So when you try to click Properties of the iSCSI adapter to modify certain settings, you see the following error:

Error:

Call "IscsiManager.QueryBoundVnics" for object "iscsiManager-##" on vCenter Server "VC-name" failed

Cause:

This event occurs when ESXi's internal iSCSI daemon becomes corrupted, requiring a cleanup of the daemon's files in ESXi.

How to resolve this:

1. Migrate all virtual machines using vMotion to ESXi hosts that are unaffected.

2. Review the existing iSCSI configuration and copy the IQN and adapter settings for your iSCSI software adapter to a text document.

3. Disable the iSCSI adapter.

4. Navigate to the /etc/vmware/vmkiscsid/ directory on your ESXi host and back up the contents of the folder to a safe location.

5. Delete the contents of /etc/vmware/vmkiscsid/

6. Write the changes to the ESXi boot bank using this command:

# backup.sh

7. Reboot the ESXi host.

8. Create a new software iSCSI adapter and configure it as per the backup you saved in step 2.

9. Add the iSCSI port bindings and targets.

That's pretty much it. You should be good to go.

VDP Backup With External Proxy

So in the previous article we saw the types of backup protocol used by VDP. We saw that when the ESXi host which is hosting the VDP appliance does not have the access to the datastore of the VM it is backing up, then the backup protocol used is Network Block Device (NBD). One way to avoid NBD is to bring the VM on to a storage which is seen by the ESXi host hosting the VDP appliance. The other method is to deploy an external proxy for the VDP appliance onto the host which is hosting the client that needs to be backed up. Please note, that once you deploy external proxy, the internal proxy will be disabled automatically. 

To add external proxy in detail, you can visit this link here.

From the below screenshot you can see that the external proxy is deployed on the host where the client that requires to be backed up resides. 

This will create a virtual machine for the proxy on the specified host.

Now, ideally, when I run the backup for this client without external proxy, it should work with the NBD protocol. However, now the proxy where the avAgent runs, can see the datastore of the client virtual machine, it will proceed with the Hot-Add protocol. 

Since avAgent runs on the proxy which talks to the MCS, the disk will not be mounted on the VDP appliance, instead it will be mounted on the external proxy virtual machine. You can see this from the below screenshot. 

vdp_ba_vma41 is the proxy virtual machine, and the client being backed up is CentOS7 and we can see the virtual disk of client is mounted on the external proxy virtual machine. 

Well, that's one of the use case of deploying external proxy.

VDP Hot-Add vs NBD

When a backup job is initiated for a virtual machine, there are two methods or protocols by which VDP performs the backup. 

The first one we have is the Hot-Add protocol, wherein the virtual disks of the virtual machine being backed up is going to be mounted on to the VDP appliance and then the backup will be performed. 

The second one is using Network Block Device (NBD) protocol, where in the virtual disk backup is done via the management network.

In the below screenshot, you can see the VM, New Virtual Machine, resides on Recovery_LUN datastore. 

From the next screenshot you can see that the VDP appliance virtual machine is residing on VDP-Datastore.

For the ESXi host where the VDP appliance resides, the storage configuration page shows that it does not have access to the Recovery_LUN datastore. This means the ESXi host hosting the VDP appliance does not see the virtual machine's datastore. 

In this scenario, where the ESXi host where the VDP appliance does not see the VMs datastore, Hot-Add protocol is not going to work and it falls back to the NBD protocol. 

Here are a few test results while using the NBD protocol:

I have a backup job called "ble" (Please ignore the name) and this contains the virtual machine named, New Virtual Machine. 

The next thing I did was, I started a manual backup job for this virtual machine. The backup job as usual initiates two tasks, create a virtual machine snapshot and a VDP: Backup Job.

When you go to Snapshot Manager for the virtual machine being backed up, you can see the snapshot created by the VDP, goes by VDP-EPOCH of backup.

And if I check the Edit Settings of the VDP appliance, I do not see the disk of the New Virtual Machine Hot-Added to the VDP appliance.

So now, the backup traffic will be flowing via the management network. So I performed a little esxtop test to check for this information. 

From the SSH session for the host which is hosting the virtual machine being backed up, I saw the following:

The virtual machine and the management network resides on the vSwitch0 with a vmnic2 uplink and we can see that the PKTTX/s is high and rising and dropping accordingly throughout the backup process. 

Next, if I SSH to the destination ESXi host where the VDP appliance is running, I see the following:

The VDP appliance is on vSwitch0 and is connected to an uplink vmnic0 and we can see the PKTRX/s is similarly high and is almost at the same rate of PKTTX/s of the virtual machine being backed up. 

Next, the observations of Hot-Add:

I am going to migrate this VM to a datastore which is seen by the ESXi host hosting the VDP appliance. 

So I migrated the VM to be backed up to the same host as my VDP appliance (As this host does not have any shared datastore) and put it on a different datastore, Suhas-Local-2. From my above screenshots you can see the Suhas-Local-2 is a datastore on my VDP hosting ESXi host. 

I then ran the backup job again, and this time, if I come to the Edit Settings of the VDP appliance I see that there is a disk added to the VDP appliance and this corresponds to the virtual disk of the VM being backed up. 

Additional information:

If you have virtual flash read cache disks (VFRC) in your environment, then those virtual machines have to be on virtual hardware 10 and above. When your VDP is on a lower version than the VMs with VFRC disks, then the backup protocol is always going to be NBD regardless if the ESXi host can see the datastore or not. 

Well, that's pretty much it!

VDP Deduplication Process

Saving the same data after every iteration of backup is not ideal because the space consumed on you storage increases rapidly. To provide a better storage for backups deduplication technology is used. What this does is, during the first initial full back, the entire contents of the virtual machine is backed up. However, the subsequent backups only save the new data or the changes that has occurred when compared to the previous iteration of backup. This is called as incremental backup. The changed data will be processed by VDP and saved where as pointer files are created to the same/unchanged data that was present in the previous backup. This saves the storage space and also increases the backup efficiency.  

Before we jump deeper into deduplication, let us have a look at the two types of deduplication we have at hand; Fixed Length (Also called as Fixed block) and Variable length (Variable block) deduplication.

This is the raw data that I have at hand right now:

"Welcome to virtuallypeculiar read abot vmware technology"

Fixed length deduplication: I am going to segment this raw data into a data-set defined by a block length of 8. Which is, 8 characters per data-set. The output will look something as:

Variable length deduplication: In this, we do not have a constant set of deduplication block length. The algorithm is going to look at the data set and set logical boundaries for deduplication length. The output will something as:

Now, on a high level basis, my backup software took a backup of the raw data which is saved on my notepad file. Since this is a first backup, the entire text data is saved on the storage using a deduplication technology. 

Next, in the raw data I have a spelling error for the word abot (about), upon noticing this, I will re-open the notepad make the necessary changes and save the file again. When the next iteration of backup runs, it is going to scan for changes in blocks. 

How fixed length deduplication deals with this?

When the new character is added, the data bits are shifted towards the right by 1. The output would be something as:

Now, in these cases there are scenarios where the shifting of data bits causes the shifted data to enter a new 8-character data-set which creates a new storage block to be occupied, just for one character. This reduces the storage efficiency when compared to variable length deduplication.

How variable length deduplication deals with this?

When the changed data is detected the variable length deduplication makes sure that the outcome of the changed data set matches the chunk size or data set size of the previous backup iteration. The output is something as:

Here the red box shows the changed data, and it is seen that it is limited to the same block whereas in fixed length it was seen till the end of the data set.
VDP is based on variable length deduplication, and using an algorithm the logical boundaries are set for the raw data.

Final note, variable length deduplication provides better storage density than fixed length as the changes in data-set is not vast.

How does VDP deduplication work?

Now, since you have a fair understanding of deduplication, we can look into how VDP handles deduplication. Please note, throughout the process, VDP uses only variable block deduplication.

Have a look at the flow chart below for the basic flow of deduplication process:

Before we get to the working of the flow chart let's have a little understanding regarding the various daemons or processes involved in this backup.

MCS (Management Console Server) This is responsible for the management of all your backup request, VDRDB database.

There are 8 internal proxies on the VDP appliance. Each proxy runs a process called avAgent. These query the MCS every 15 seconds for incoming job requests. Once the avAgent receives the backup request it in turn calls the avVcbImage

The avVcbImage is responsible for enabling, browsing backing up and restoring the disks.

The avVvcbImage in turn calls the avTar which is the primary process for backup and restore.


The entire deduplication process occurs inside the VDP appliance. When the backup request comes in, the first check is done on the client side, where the appliance determines if this virtual machine has been backed up or not. The .ctk file created due to CBT feature when a backup is taken records all the changed sector information since the previous backup. When the appliance scans for this, and if the ctk file determines the changes, only those changed data is sent further to the Sticky Byte Factoring. If older data is present, it is going to create pointer files and will be excluded from Sticky byte factoring.

In Sticky Byte Factoring:

The avTar running here is responsible for breaking down the raw data input that we received earlier into data chunks. The data set that is an outcome of this will be anywhere from 1 KB to 64 KB and will average out on a 24 KB set.

The earlier example we considered for variable block deduplication, let's use that data set, represent that in terms of KB of data and re-review the deduplication process.

So here, in the first full backup, the raw data is divided into variable length blocks using VDP algorithm. It produces a set of data chunks anywhere between 1 and 64K. Now, the data in the first two blocks have changed after the backup was performed. Now, in the next iteration of backup, the sticky byte factoring re-syncs the block so the output of new dataset matches the chunks of the previous dataset. So, no matter where the data has changed the avTar creates chunks to match the previous chunk size.

Compression:

Once the sticky byte factoring divides the raw data into chunks, these will be compressed. The compression ratio will be anywhere between 30 and 50 percent and data that is not favourable for compression will be omitted to prevent performance impact.

Hashing:

The compressed data is then hashed using SHA-1 algorithm. And the hashed data will always output a 20 byte data string. This hash data is unique to each block and serves as a reference for comparison to check if the previous backup has a similar has. If yes, then the similar hashed data are excluded from backup. Hashing does not convert data into hashes, it rather creates hashes for each data block. So at the end of Hashing, you will have your data chunks and hashes corresponding to it. If the hashes are not found in the hash cache, then the cache is updated with the new hash. 

The above hashes are called as atomic hashes, further the atomic hashes are combined to form composites. The composites are further combined to form composite hashes. This process is continued until one single root hash is created.
So in the end, we have the actual data stripes. the atomic hashes, the composite hashes and the root hash all stored in their own locations on the VDP storage disks.

Hope this was helpful. If you have any questions, please feel free to reply.


All images are copyrights of virtuallyPeculiar.com

Cannot Perform A Backup With VDP 5.1

So while performing a backup of any virtual machines in my environment on VDP, the backup task used to fail. So, I created one backup job, with one virtual machine in it and ran the backup task. The create snapshot task completes successfully and you can see the VDP snapshot in the snapshot manager of the virtual machine. Once the snapshot is taken, the next task initiated will be the VDP backup task, and this fails immediately (no percentage completed). It states it failed with miscellaneous errors. 

When you open a SSH to the VDP appliance and browse the backup-job log, from the directory, /usr/local/avamarclient/var-proxy-N, I see the following logging:

2016-04-20 08:28:50 avvcbimage FATAL <5889>: Fatal signal 6 in pid 553

2016/04/20-15:28:50.11148 []  FATAL ERROR: <0001> Fatal signal 6

2016/04/20-15:28:50.11161 []  | 00000000005f39f1

2016/04/20-15:28:50.11162 []  | 00000000005f46b7

2016/04/20-15:28:50.11163 []  | 00000000005f5bdb

2016/04/20-15:28:50.11164 []  | 00000000005f5cce

2016/04/20-15:28:50.11168 []  | 000000000058b800

2016/04/20-15:28:50.11169 []  | 00007ff23c7c16a0

2016/04/20-15:28:50.11170 []  | 00007ff23b0797a9

2016/04/20-15:28:50.11170 []  | 00007ff2308a8900

2016/04/20-15:28:50.11171 []  | 00007ff23087b9ac

2016/04/20-15:28:50.11172 []  | 00007ff2306bf442

2016/04/20-15:28:50.11173 []  | 00007ff2362c472e

2016/04/20-15:28:50.11174 []  | 00007ff2362cff4c

2016/04/20-15:28:50.11174 []  | 00007ff230898d17

2016/04/20-15:28:50.11175 []  | 00007ff230894883

2016/04/20-15:28:50.11176 []  | 00007ff23c7b9696

2016/04/20-15:28:50.11177 []  | 00007ff23b07cd7d

2016/04/20-15:28:50.11184 []  ERROR: <0316> handlefatal exiting thread pid=553, sig=6

2016-04-20 08:28:50 avvcbimage Error <5891>: handlefatal: exiting thread pid=553, sig=6

2016-04-20 08:28:50 avvcbimage Info <16041>: VDDK:2016-04-20T08:28:50.113-07:00 [7FF22EE0D700 panic 'Default']

2016-04-20 08:28:50 avvcbimage Info <16041>: VDDK:-->

2016-04-20 08:28:50 avvcbimage Info <16041>: VDDK:--> Panic: Assert Failed: "_lockToken != __null" @ /build/mts/release/bora-774844/bora/lib/vcbLib/hotAdd.cpp:638

Here the appliance is failing to lock the VMDK which it has to backup and entering a panic state causing the backup to fail. 

The 5.1 VDP has a VDDK version of 5.1. VDDK is a set of C++ libraries that is used to access and create virtual disk. The version of VDDK can be found out from the backup job log (Just search for VDDK and the first few lines will define the version)

You will see the entry for VDDK version as follow (This was on my 6.1.2 VDP)

2016-04-18T20:00:15.978+07:00 avvcbimage Info <16041>: VDDK:VMware VixDiskLib (6.0) Release build-2498720

Now, back to the failed backup. This is a known issue with 5.1 VDDK version and this can be verified from the 5.5.1 (5.5 Update 1) VDDK release notes. (Search for the section: Assert failure causes exit when HotAdd cannot acquire lock)

Upgrade your VDP to 5.8 and we should be good to go as it updates the underlying VDDK version. Re-run a backup after the upgrade and check the backup logs again and you can see the new version of VDDK release and it's corresponding build number.

Changing DNS for VDP 6.1.2 post deployment.

Short article for a quick reference to update the DNS name for VDP appliance. While working on a case today I had to change the DNS name from vdp.vapp.local to vdpnew.vapp.local

From the below screenshots, you can see that the "hostname" from VDP appliance console shows the original DNS for the appliance.

The ping to the DNS resolves to the current IP of the appliance.

Now, to update this record:

1. Update the DNS record from your DNS manager. 

2. Now login to the vdp configure page which can be found at the below location:

https://vdp-ip:8543/vdp-configure

3. Here under the host-name section we will see the old VDP appliance DNS entry. This has to be updated. Click the gear icon and select Network Settings.

4. You will come across the above screen and update the host-name section with the new host-name that was given for the appliance and click Next and Finish. 

5. Restart the Guest for VDP appliance and we should be good to go from there. 

You can now see the updated DNS for VDP.

That's it!