Upgrading vSphere Replication From 6.5 To 8.1

With the release of vSphere Replication 8.1, the upgrade path is not how it was earlier. The 8.1 vR server now hosts a PhotonOS and the upgrade is similar to a vCenter migration. In this case, you will deploy a new 8.1 vR server via the OVF template with a temporary IP and then follow a series of upgrade / migrate steps to transfer data from the old vR server to the new one.

1. Proceed with the regular deployment of the vSphere replication appliance, where you download the 8.1 ISO, mount it on a windows server and choose the support.vmdk, system.vmdk, certificate, manifest and the OVF file for deployment. A temporary IP is needed for the appliance to be on network.

2. Once the deployment is done, power on the 8.1 appliance and open a VM console. During the boot you will be presented with the below options.

The 192.168.1.110 is my 6.5 vSphere Replication appliance and it was already registered to the vCenter server. Select the Option 3 to proceed with the Upgrade.

NOTE: For Bad Exit Code 1 error during upgrade, refer this article here.

3. Provide in the root password of the old replication server to proceed.

4. The upgrade process begins to install the necessary RPMs. This might take about 10 minutes to complete.

5. You will then be prompted to enter the SSO user name of the corresponding vCenter this vR is registered to and it's password.

6.  Post a few configuration progress in the window, the upgrade is done and you will be presented with the 8.1 banner page.

That should be it. Hope this helps!

Upgrading vCenter Appliance From 6.5 to 6.7

So as you know, vSphere 6.7 is now GA and this article will speak about upgrading an embedded PSC deployment of 6.5 vCenter appliance to 6.7. Once you download the 6.7 VCSA ISO installer
mount the ISO on a local windows machine and then you can use the ui installer for windows to begin the upgrade phase.

You will be presented with the below choices:

We will be going with the Upgrade option. The upgrade is going to be like the earlier path wherein the process will deploy a new 6.7 VCSA and perform a data and configuration migration from the older 6.5 appliance and then power down the old server when the upgrade is successful.

Accept the EULA to proceed further.

In the next step we will connect to the source appliance so provide in the IP/FQDN of the source 6.5 vCenter server.

Once the Connect To Source goes through you will then be asked to enter the SSO details and the ESX details where the 6.5 vCSA is running.

Then the next step is to provide information about the target appliance, the 6.7 appliance. You will select the ESX where the target appliance should be deployed.

Then provide the inventory display name for the target vCenter 6.7 along with the a root password.

Select the appliance deployment size for the target server. Make sure this matches or is greater than the source 6.7 server.

Then select the datastore where the target appliance should reside.

Next, we will provide a set of temporary network details for the 6.7 appliance. The appliance will inherit the old 6.5 network configuration post a successful migration.

Review the details and Finish the begin the Stage 1 deployment process.

Once the Stage 1 is done, you can Continue to proceed further with the Stage 2

In the Stage 2 we will be performing a data copy from the source vCenter appliance to the deployed target from Stage 1

Provide in the details to connect to the source vCenter server.

Select the type of data to be copied over to the destination vCenter server. In my case, I just want to migrate the configuration data.

Join the CEIP and proceed further

Review the details and Finish to begin the data copy.

The source vCenter will be shutdown post the data copy.

The data migration will take a while to complete and is in 3 stages.

And that's it. If all goes well, the migration is complete and you can access your new vCenter from the URL.

Hope this helps.

Creating A VMFS Volume From Command Line

One of the alternate methods to formatting a new VMFS volume from the GUI is to create the same from the SSH of the ESXi host.

The process is quite simple and you can follow them as mentioned below:

1. Make sure the device is presented to the ESX and visible. If not, perform a Rescan Storage and check if the device is visible.

2. You can get the device identifier from the SSH of the ESX by navigating to:

# cd /vmfs/devices/disks

In my case, the device I was interested was mpx.vmhba1:C0:T3:L0

3. Next, we need to create a partition on this device and we no longer use fdisk for ESX as this is deprecated. So we will use partedUtil

So, we will create a partition (Number=1) at an offset of 128. The partition identifier is 0xfb which is a VMFS partition. 0xfb = 251. Along with this we will specify the ending sector.

To calculate ending sector:
The disk has 512 bytes per sector. In my case the device is 12 GB.
So number of bytes is 12884901887.99998
Dividing this by 512 is 25165824 sectors.

Do not use the complete sector value as it might complain out of bound sector value, so use one number less.

The command would then be:

# partedUtil set /vmfs/devices/disks/device-name "1 128 <ending-sector> 251 0"

Sample command:

# partedUtil set /vmfs/devices/disks/mpx.vmhba1:C0:T3:L0 "1 128 25165823 251 0"

A successful output would be:
0 0 0 0
1 128 25165823 251 0

4. Next, you format a VMFS volume using the vmkfstools -C command. 

The command would be:

# vmkfstools -C <vmfs-version> -b <block-size> -S <name-of-datastore> /vmfs/devices/disks/<device-name>:<partition-number> 

So the command for me would be (For a VMFS5 partition with 1 mb block size)

# vmkfstools -C vmfs5 -b 1m -S Test /vmfs/devices/disks/mpx.vmhba1:C0:T3:L0:1

A successful output would be:

Checking if remote hosts are using this device as a valid file system. This may take a few seconds...

Creating vmfs5 file system on "mpx.vmhba1:C0:T3:L0:1" with blockSize 1048576 and volume label "Test".

Successfully created new volume: 5aa7d4e8-1e99a608-f609-000c292cd901

Now, back in the GUI just do a refresh on the storage section and this volume is visible for the host. 

Hope this helps!

VDP Expired MCSSL, Reports 7778, 7779, 7780, 7781, 9443 As Vulnerable In Nessus Scan

In one of my case, there was a report that 7778, 7779, 7780, 7781, 9443 where reported as vulnerable on VDP 6.1.6. All these are MCS java based ports and you can confirm them by running:

# netstat -nlp | grep <enter-port>

To check your MCS SSL validity perform the below commands:

# /usr/java/default/bin/keytool -list -keystore /usr/local/avamar/lib/rmi_ssl_keystore -storepass changeme

The output:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 2 entries

mcssl, Feb 1, 2008, PrivateKeyEntry,

Certificate fingerprint (SHA1): F1:61:A7:FE:36:A9:E9:7E:DB:92:AE:89:05:52:13:B6:3C:FA:55:A7

vcenterrootca, Jan 8, 2018, trustedCertEntry,

Certificate fingerprint (SHA1): F0:46:B4:00:B8:52:24:6E:A2:94:6B:17:CE:83:23:49:54:9A:3A:49

Then export the cert to root directory:

# /usr/java/default/bin/keytool -exportcert -v -alias mcssl -keystore /usr/local/avamar/lib/rmi_ssl_keystore -storepass changeme -file /root/mcssl.cer -rfc

The output:

Certificate stored in file </root/mcssl.cer>

Then read the certificate:

# /usr/java/default/bin/keytool -printcert -v -file /root/mcssl.cer

The output:

Owner: CN=Administrator, OU=Avamar, O=EMC, L=Irvine, ST=California, C=US

Issuer: CN=Administrator, OU=Avamar, O=EMC, L=Irvine, ST=California, C=US

Serial number: 47a25760

Valid from: Fri Feb 01 00:18:56 CET 2008 until: Mon Jan 29 00:18:56 CET 2018

Certificate fingerprints:

MD5: 61:42:FC:CD:FC:CB:6E:59:CC:48:5E:D9:71:05:F0:B4

SHA1: F1:61:A7:FE:36:A9:E9:7E:DB:92:AE:89:05:52:13:B6:3C:FA:55:A7

SHA256: B4:E6:71:77:58:9B:58:64:E2:F7:3A:A0:2A:07:F8:7B:2E:CA:1B:22:2B:C3:98:A8:90:F8:D8:7A:8E:0A:EE:F9

Signature algorithm name: SHA1withDSA

Version: 1

Due to this expired cert, the java ports are vulnerable. To fix this, you will have to regenerate the certs. The process would be:

1. Backup existing keystore:

# cp /usr/local/avamar/lib/rmi_ssl_keystore ~root/rmi_ssl_keystore_backup-`date -I`

2. Regenerate the mcssl:

# /usr/java/latest/bin/keytool -genkeypair -v -alias mcssl -keyalg RSA -sigalg SHA256withRSA -keystore /usr/local/avamar/lib/rmi_ssl_keystore -storepass changeme -keypass changeme -validity 3650 -dname "CN=`hostname -f`, OU=Avamar, O=EMC, L=Irvine, S=California, C=US" -keysize 2048

Generates a SHA256 SSL which is valid for 10 years. 

3. Update the permissions on the rmi_ssl_keystore

# chmod 444 /usr/local/avamar/lib/rmi_ssl_keystore

4. Update owners for the keystore:

# chown root:admin /usr/local/avamar/lib/rmi_ssl_keystore

5. Switch to admin mode and restart MCS:

# mcserver.sh --stop 

# mcserver.sh --start --verbose

6. Verify all vCenter Connections are OK:

# mccli server show-services

That should be it. Now when you re-run the scan these ports are no longer vulnerable. 

Hope this helps!

vSphere Replication 6.5.1 With vRealize Orchestrator 7.3

Here we will be looking into how to configure and use vSphere replication with vRealize Orchestrator. The version of my setup is:

vCenter Appliance 6.5 U1
vSphere Replication 6.5.1
vRealize Orchestrator 7.3

In brief, deploy the vRealize Orchestrator OVA template. Then navigate to "https://<vro-fqdn>:8283/vco-controlcenter/" to begin the configuration.

I have a standalone Orchestrator deployment with vSphere Authentication mode.

SSO user name and password is required to complete the registration. A restart of vRO would be needed to complete the configuration.

Next, download the vSphere Replication vmoapp file from this link here.

To install this file, click on the Manage Plugins tab in the Orchestrator control center and browse for the downloaded vmoapp file.

Then accept the EULA to Install the Plugin.

If prompted, click Save Changes and this should show the vR plugin is available and enabled in the plugin page.

Next, register the vCenter Site for the replication server using the below "Register VC Site" Workflow. All the next tasks are done from the Orchestrator client.

Once done, you can verify the vSphere Replication site is now visible under Administer mode of vRO.

Next, we will configure replication for one virtual machine. With the Run mode execute the "Configure Replication" workflow.

The Site (source) will be selected first.

Selecting virtual machine will be the next task.

Target site vR selection will be next. I am replicating within the same vCenter, so the source and target vR site is the same machine.

Next, we will select the target datastore where the replicated files should reside.

Lastly, we will choose the RPO and other required parameters to complete the replication task and click Submit.

Finally, you can see the VM under Outgoing Replication tab for vCenter.

That's pretty much it!

Bash Script To Extract vSphere Replication Job Information

Below is one bash script that extracts information about replication for configured VMs. It displays, the name of the virtual machine, if yes or no for quiesce Guest OS and Network Compression. Then it tabulates RPO (in minutes) as "bc" is unsupported on vR SUSE to perform hour floating calculations and then the datastore MoRef ID.

The complete updated script can be accessed from my GitHub Repo:
https://github.com/happycow92/shellscripts/blob/master/vR-jobs.sh

As and when I add more or reformat the information the script in the link will be updated.

#!/bin/bash
	clear
	echo -e " -----------------------------------------------------------------------------------------------------------"
	echo -e "| Virtual Machine | Network Compression | Quiesce | RPO | Datastore MoRef ID |"
	echo -e " -----------------------------------------------------------------------------------------------------------"
	cd /opt/vmware/vpostgres/9.3/bin
	./psql -U vrmsdb << EOF
	\o /tmp/info.txt
	select name from groupentity;
	select networkcompressionenabled from groupentity;
	select rpo from groupentity;
	select quiesceguestenabled from groupentity;
	select configfilesdatastoremoid from virtualmachineentity;
	EOF
	cd /tmp
	name_array=($(awk '/name/{i=1;next}/ro*/{i=0}{if (i==1){i++;next}}i' info.txt))
	quiesce_array=($(awk '/networkcompressionenabled/{i=1;next}/ro*/{i=0}{if (i==1){i++;next}}i' info.txt))
	compression_array=($(awk '/quiesceguestenabled/{i=1;next}/ro*/{i=0}{if (i==1){i++;next}}i' info.txt))
	rpo_array=($(awk '/rpo/{i=1;next}/ro*/{i=0}{if (i==1){i++;next}}i' info.txt))
	datastore_array=($(awk '/configfilesdatastoremoid/{i=1;next}/ro/{i=0} {if (i==1){i++;next}}i' info.txt))
	length=${#name_array[@]}
	for ((i=0;i<$length;i++));
	do
	printf "| %-32s | %-23s | %-10s | %-10s| %-20s|\n" "${name_array[$i]}" "${quiesce_array[$i]}" "${compression_array[$i]}" "${rpo_array[$i]}" "${datastore_array[$i]}"
	done
	rm -f info.txt
	echo && echo

For any questions, do let me know. Hope this helps. Thanks.

Bash Script To Determine Retired Clients.

While in VDP you have a built in feature for unprotected VMs (That is VMs not added to VDP backup job) you might need a script to determine if VMs are missing from a backup job.

The script has a simple algorithm:
> The first time it runs it creates a file to gather all the protected client list
> The next time it runs it will check what is missing since the last protect client list.
> New added VMs will not be considered as Missing, however on Next iteration of script execution it will run a check to see if the new clients are missing.
> If you remove the first generated file for protected list post your second execution, then the third iteration will be void as it will generate a new protected client list.

The script has an email feature to send the output to a mailing address. If you want to exclude this, then discard line-21 to line-32. If you want to run the script as a cronjob, you can add it to crontab -e, but you cannot have manual email address input running in the script. You will have to create a constant for your email address and call it in the EOF.

The script can be accessed from my repository here:
https://github.com/happycow92/shellscripts/blob/master/missing-client.sh

The code {}

#!/bin/bash
	IFS=$(echo -en "\n\b")
	FILE=/tmp/protected_client.txt
	if [ ! -f $FILE ]
	then
	client_list=$(mccli client show --recursive=true | grep -i /$(cat /usr/local/vdr/etc/vcenterinfo.cfg | grep vcenter-hostname | cut -d '=' -f 2)/VirtualMachines | awk -F/ '{print $(NF-2)}')
	echo "$client_list" &> /tmp/protected_client.txt
	sort /tmp/protected_client.txt -o /tmp/protected_client.txt
	else
	new_list=$(mccli client show --recursive=true | grep -i /$(cat /usr/local/vdr/etc/vcenterinfo.cfg | grep vcenter-hostname | cut -d '=' -f 2)/VirtualMachines | awk -F/ '{print $(NF-2)}')
	echo "$new_list" &> /tmp/new_list.txt
	sort /tmp/new_list.txt -o /tmp/new_list.txt
	missing=$(comm -3 /tmp/protected_client.txt /tmp/new_list.txt | sed 's/^ *//g')
	if [ -z "$missing" ]
	then
	printf "\nNo Client's missing\n"
	else
	printf "\nMissing Client is:\n" | tee -a /tmp/email_list.txt
	printf "$missing\n\n" | tee -a /tmp/email_list.txt
	printf "Emailing the list\n"
	FILE=/tmp/email_list.txt
	read -p "Enter Your Email: " TO
	FROM=admin@$(hostname)
	(cat - $FILE)<< EOF | /usr/sbin/sendmail -f $FROM -t $TO
	Subject: Missing VMs from Jobs
	To: $TO
	EOF
	sleep 2s
	printf "\nEmail Sent. Exiting Script\n\n"
	fi
	rm /tmp/new_list.txt
	rm -f /tmp/email_list.txt
	fi

Feel free to reply for any issues. Hope this helps!