Removing A Stale Inaccessible NFS Datastore From vCenter

I had recently run into an issue, where one of the NFS datastores coming off a NAS share from one of my backup appliances (Rubrik) had gone inaccessible. 

I did a bit of look around to see what had happened, and I noticed that the NAS share was destroyed off from the backup end, in a rather unclean way which left some stale entries on the VMware end of it. 

The highlighted datastore was one of the datastore in question and I had to get rid of the entry from the vCenter and the ESX host. The VM residing on it was no longer needed for salvage so it was all good to let go of any data residing on it since the backing share was destroyed anyways. 

First, I had to disassociate the datastore from the ESX host, which was straightforward. From an SSH session to ESXi, all I had to do was run:

esxcli storage nfs remove -v <datastore_name> 

This removed the host relationship with the datastore and the UI displayed something like below

Now you cannot do an Unmount datastore in such a state is because there are no more ESX hosts associated with this datastore and this needed to be cleaned up manually from the vCenter Postgres database. 

Before making any changes to VCDB, ensure you have a vCenter backup, snapshot or whatever is necessary to revert back to a working state if something goes wrong. 

Stop the vmware-vpxd service using

service-control --stop vmware-vpxd

Connect to the VCDB using 

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Then run the below query, replacing the datastore name that you want to clean up 

SELECT id from vpx_entity where name='rubrik_9c65faa5cbe34dd6a9a796f1b4d12c20';

You will then see an output such as

VCDB=# SELECT id from vpx_entity where name='rubrik_9c65faa5cbe34dd6a9a796f1b4d12c20';
id
-------
 10369
(1 row)

Using this ds_id you can see the assignment 

VCDB=# select * from vpx_ds_assignment where ds_id=10369;
 ds_id | entity_id | accessible | mount_path | mount_id | mount_mode | mounted
-------+-----------+------------+------------+----------+------------+---------
10369 |     10370 |            |            |          |            |
(1 row)

Then all you need to do is clean up the entry from the below 3 tables using

VCDB=# delete from vpx_ds_assignment where ds_id=10369;
VCDB=# delete from vpx_datastore where id=10369;
VCDB=# delete from vpx_vm_ds_space where ds_id=10369;

Replace the IDs accordingly

Once done, finally start the vmware-vpxd service using

service-control --start vmware-vpxd

The stale datastore entry should now no longer be visible

Hope this helps.