vSphere 6.0 Web Client Does Not Load After An IP Change

Recently, I was tasked to change the IP address of my lab environment from 192.x range to a 10.x range. The deployment of vCenter was an appliance with an external Platform Services Controller.
All the deployment and configuration was done with FQDN so changing IP address was not an issue.

I changed the IP of vCenter and PSC node from the Web Management Interface and restarted both these nodes. However, post the restart the web client was not loading up and I was seeing the below message:

When I checked service --status-all on the vCenter node, I saw a bunch of services were not running. I started these services manually from the CLI using this KB here

However, there was no luck. If we check the hosts file on both PSC and vCenter, it is still using the old IP address.

All we have to do here is, edit this file and change the IP from the old address to the new one. Post this, save the file and restart both the PSC and vCenter nodes.

Now, all the services in vCenter will start automatically and the web client will load without issues.

Hope this helps.

VDP Upgrade To 6.1.3 Fails: Installation Of The Package Stalled

vSphere Data Protection 6.1.3 is a pretty stable release in the 6.x chain. However, there was an issue getting to this version from a prior release. This issue is specifically seen while upgrading VDP from 6.1 to 6.1.3. The upgrade fails approximately around 60 percent, with an error "Installation of the package stalled" and it prompts you to revert to the snapshot taken prior to the upgrade.

The avinstaller.log has the following:

INFO: Working on task: Starting MCS (74 of 124) id: 10312
Dec 12, 2016 1:07:32 PM com.avamar.avinstaller.monitor.PollMessageSender sendMessage
INFO: PollMessageSender sent: <UpdateMessage><Progress>60</Progress><TaskName>Starting MCS (74 of 124)</TaskName><ProcessKey>VSphereDataProtection20161020Oct101476964022</ProcessKey><ProcessInstanceId>VSphereDataProtection20161020Oct101476964022.10007</ProcessInstanceId><TaskId>10312</TaskId><Timestamp>2016/12/12-13:07:32.00489</Timestamp></UpdateMessage>
INFO: Package Location in process: /data01/avamar/repo/temp/vSphereDataProtection-6.1.3.avp_1481542766965
<UpdateMessage><ProcessKey>VSphereDataProtection20161020Oct101476964022</ProcessKey><ProcessInstanceId>VSphereDataProtection20161020Oct101476964022.10007</ProcessInstanceId><Timestamp>2016/12/12-13:07:38.00626</Timestamp><Content>Starting MCS ...</Content></UpdateMessage>
Dec 12, 2016 1:07:38 PM com.avamar.avinstaller.process.TaskTimeManager resetTimer
INFO: startTime: 1481544458653
Dec 12, 2016 1:08:38 PM com.avamar.avinstaller.monitor.MessageListener doPost
INFO: MessageListener receiving message
Dec 12, 2016 1:08:38 PM com.avamar.avinstaller.monitor.MessageListener doPost
INFO: MessageListener forwarding message: <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<UpdateMessage><Content><![CDATA["mcserver.sh --start", exit status=1 (error)]]></Content></UpdateMessage>
Dec 12, 2016 1:08:38 PM com.avamar.avinstaller.process.MessageReceiver receiveMessage
INFO: MessageReceiver receive message: <UpdateMessage><Timestamp>2016/12/12-13:08:38.00890</Timestamp><Content>"mcserver.sh --start", exit status=1 (error)</Content></UpdateMessage>
Dec 12, 2016 1:08:38 PM com.avamar.avinstaller.monitor.PollMessageSender sendMessage
INFO: PollMessageSender sent: <UpdateMessage><ProcessKey>VSphereDataProtection20161020Oct101476964022</ProcessKey><ProcessInstanceId>VSphereDataProtection20161020Oct101476964022.10007</ProcessInstanceId><Timestamp>2016/12/12-13:08:38.00890</Timestamp><Content>"mcserver.sh --start", exit status=1 (error)</Content></UpdateMessage>
Dec 12, 2016 1:08:38 PM com.avamar.avinstaller.process.TaskTimeManager resetTimer
INFO: startTime: 1481544518935
Dec 12, 2016 1:08:43 PM com.avamar.avinstaller.monitor.MessageListener doPost
INFO: MessageListener forwarding message: <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<UpdateMessage><Content><![CDATA[----->start MCS failed - operation failed]]></Content></UpdateMessage>
<UpdateMessage><ProcessKey>VSphereDataProtection20161020Oct101476964022</ProcessKey><ProcessInstanceId>VSphereDataProtection20161020Oct101476964022.10007</ProcessInstanceId><Timestamp>2016/12/12-13:08:43.00689</Timestamp><Content>-----&gt;start MCS failed - operation failed</Content></UpdateMessage>

Dec 12, 2016 1:08:44 PM com.avamar.avinstaller.process.handler.ServerScriptHandler$TaskProcessor run
WARNING: From err out: 12/12 13:08:38 error: "mcserver.sh --start", exit status=1 (error)
12/12 13:08:43 error: ----->start MCS failed - operation failed

Dec 12, 2016 1:50:04 PM com.avamar.avinstaller.util.FlushTimer run
INFO: Flush exit stderr:
Dec 12, 2016 2:50:00 PM com.avamar.avinstaller.util.FlushTimer run
INFO: Flush command: /usr/local/avamar/bin/avinstaller.pl --flush
Dec 12, 2016 2:50:03 PM com.avamar.avinstaller.util.FlushTimer run
INFO: Flush exit code: 1
Dec 12, 2016 2:50:03 PM com.avamar.avinstaller.util.FlushTimer run
INFO: Flush exit stdout: Flushing Avinstaller...
INFO: Flushing AvInstaller.
INFO: AvInstaller flushed.

Unfortunately there is no fix available for this issue. This is a known issue while upgrading from 6.1 to 6.1.3. See the release notes here

The workaround is to upgrade from 6.1 to 6.1.2 and then upgrade the 6.1.2 to 6.1.3
If this workaround fails, please raise a ticket with VMware Support. I cannot provide the resolution here due to confidentiality.

Hope this helps.

Part 1: Deploying And Configuring vSphere Replication 6.x

If you are using Site Recovery Manager, you would be using a replication technique to replicate the VM data from a protected to a recovery site. There are two ways to get this done. The first one is using a Host Based Replication and the second is Array Based Replication.

vSphere Replication is a Host Based Replication to perform VM replication via the VMkernel. In this case, I will have two vCenters. One in protected and other in recovery site. The deployment for replication is completely from the web client from the 6.x version onward.

Right click any one of the host and select Deploy OVF Template

Browse the local file location to access the .ovf file of this appliance. Select this file and click Open.
Once this file is loaded, click Next

You will be presented with the OVF details regarding the Vendor and the version of replication being deployed. Click Next

Accept EULA and click Next

Provide a Name for the replication appliance a destination folder for deployment and then click Next.

You can switch between 2 or 4 vCPU for this appliance. I will stick to the default. Click Next.

Select a Datastore to deploy this appliance on and choose an appropriate disk provisioning. Click Next.

Select the network where this appliance should reside and IP Protocol and allocation type. If the type is Static - Manual, then enter the Gateway, DNS and Subnet details.

Enter the root password for your vR and provide the NTP server details. Expand Networking Properties and enter the IP for this appliance.

Review the changes and then complete the OVF deployment.

Post the deployment of the appliance, you will have to configure this to the vCenter. To do this, go to the management URL of the vR appliance.

https://vR-IP:5480

Login with the root credentials that was setup during the deployment. Here, there are few details to be filled:

For the LookupService Address: If your vCenter is embedded PSC deployment, then enter the FQDN of the vCenter Server. If the vCenter is an external PSC deployment, then enter the FQDN of the PSC appliance. Enter the SSO user name and the password for the same.

Click Save and Restart Service. If you are requested to accept certificate information from vCenter / PSC, accept it. Post this, the replication appliance will be configured for your vCenter.

You should be seeing this in your vR plugin in web client. I have two appliances is because one is for protected site and the other is for recovery site. 

With this, the deployment and configuration of vR to vCenter is completed.

Part 2: Configure replication sites and configuring replication for a VM.

vSphere Client Console Does Not Display Full Screen

So, while opening a console for any virtual machine from one particular workstation, the console display is not sized correctly. Below is the screenshot of the display.

This was seen on only one machine, any user sessions and a reinstall of vSphere Client did not help to resolve this. And this was seen on a Windows 10 machine.

To resolve this, you will have to disable Display Scaling.

1. Right click the vSphere Client icon and select Properties
2. Click Compatibility
3. Check box, Disable display scaling on high DPI Settings.
4. Apply Settings
5. Reload the vSphere Client session.

Now, the console session should populate the full screen.

Hope this helps.

SLACK Channel for VCAP Study Group

New Slack Channel Created for VCAP Study Group

Please comment your Email address to join the Group.

Channels available

#VCAP6-DCV Deploy

#VCAP6-DCV Design

#VCAP6-CMA Deploy

#VCAP6-CMA Design

#VCAP6-DTM Deploy

#VCAP6-DTM Design

#VCIX-NV

ISO Package Not Available During VDP Upgrade From 6.1

When you try upgrading the vSphere Data Protection Appliance from 6.1 to 6.1.1 / 6.1.2 / 6.1.3 the ISO package might not be detected. Once you mount the ISO and go to the vdp-configure page, the package is not seen and you will see the below message:

To upgrade you VDP appliance please connect a valid upgrade ISO image to the appliance

There is a known issue with the ISO detection while upgrading from 6.1 to the current latest release. To fix this, try the steps in the below order:

1. Login to SSH of the VDP appliance and run the below command to view if the ISO is mounted successfully:

# df -h

You should see this following mount:

/dev/sr0 /mnt/auto/cdrom

If this is not seen, run the below command to manually mount the ISO:

mount /dev/sr0 /mnt/auto/cdrom

Run the df -h command again to verify the mount point is now seen. Once this is true, log back into the vdp-configure page and go to Upgrade tab to verify if the ISO package is now detected. 

If not, then proceed to Step 2

2. Patch the VDP appliance with the ISO detection patch. 

Download this patch by clicking this link here. 

Once this is downloaded, perform the below steps to patch the appliance.

>> Using WinSCP copy the .gz file to a /tmp directory in the VDP appliance

>> Run the below command to extract the file:

tar -zxvf VDP61_Iso_Hotfix.tar.gz

>> Provide execute permissions to the shell script

chmod a+x VDP61_Iso_Hotfix.sh

>> Run the patch script using the below command:

./VDP61_Iso_Hotfix.sh

The script detects if the appliance is 6.1 and patches if yes. If the appliance is not 6.1 the script exits without making any changes.

Post this, log back into the vdp-configure page and the package would be detected and you will be prompted to initiate the upgrade.

Unable To Expand VDP Storage: There Are Incorrect Number Of Disks Associated With VDP Appliance

There have been few cases where I have been working on lately where there were issues expanding data storage on VDP appliance. In the vdp-configure page if you select the Expand Storage in the Storage section, you receive the error:

VDP: There are incorrect number of disks associated with VDP appliance.

This is due to incorrect update for numberOfDisk parameter in the vdr-configuration.xml file. This file should be updated if any changes are made to the VDP in the configuration page, which includes the deployment and expansion tasks. 

In my case, I had a 512 GB deployment of VDP, this creates 3 data0? partitions of 256 GB each. Which means the numberOfDisk parameter should be 3. However, in the vdr-configuration.xml file, the value was 2. Below is the snippet:

<numberOfDisk>2</numberOfDisk>

To fix this, edit the vdr-configuration.xml file located in /usr/local/vdr/etc and change this parameter to the number of data drives (do not include the OS drive) present on your VDP appliance. In my case the total number of data drives were 3. 

Save the file after editing, and re-run the expand storage task and this error should not be presented again.

Hope this helps.