I was contacted the other day with an issue where someone was trying to create a cluster in a box, and they were running into an issue. When trying to power on the VM with the shared, eager-zeroed-thick disk that was created they were running into an error message, saying that they did not have permission to access the disk.
I checked to see if the VM settings were correct and that the SCSI adapter was set to a different controller with the correct disk sharing options. It was.
I enquired how was the eager zeroed disk created, and that was also done correctly. The admin logged into the ESX host and created the disk with the following command on the datastore.
sudo vmkfstools -c 60G -d eagerzeroedthick /vmfs/volumes/IT_DBM_QT/shared/shared.vmdk
I logged into the host and noticed something weird. The disk was created in a folder called shared but the permissions on the folder and the files were not correct. they were set with owner:group which was not root:root the way it should have been.
From the ESX host
drwxrwxrwx 1 root root 4096 Nov 1 14:43 .
drwxr-xr-x 1 root root 512 Nov 1 15:23 ..
drwxr-xr-x 1 65534 65534 4096 Sep 20 14:31 md1
drwxr-xr-x 1 65534 65534 4096 Oct 27 11:23 md2
drwxr-xr-x 1 65534 65534 4096 Sep 19 16:21 mich_bi
drwxr-xr-x 1 65534 65534 4096 Oct 14 18:04 oem11
drwxr-xr-x 1 65534 65534 4096 Sep 16 10:20 orabi11
drwxr-xr-x 1 65534 65534 4096 Oct 3 12:09 rh5.5-m2.kickstart
drwxr-xr-x 1 65534 65534 4096 Sep 12 12:36 rh5-dg1
drwxr-xr-x 1 65534 65534 4096 Oct 14 16:53 rh5-dg1_1
drwxr-xr-x 1 65534 65534 4096 Sep 12 15:46 rh5-dg1_2
drwxr-xr-x 1 65534 65534 4096 Sep 13 11:05 rh5-dg2
drwxr-xr-x 1 65534 65534 4096 Sep 12 15:08 rh5-dg2_1
drwxr-xr-x 1 65534 65534 4096 Nov 1 15:07 rh5-rac1
drwxr-xr-x 1 65534 65534 4096 Nov 1 15:09 rh5-rac2
drwxrwxr-x 1 admin admin 4096 Nov 1 14:43 shared
drwxrwxrwx 1 root root 4096 Nov 1 12:00 .snapshot
Now the listing above shows that the owner was 65534:65534 which was not right.
Now who is user 65534? I tried to cat /etc/passwd | grep 65534 but did not find anything there.
I have the NFS datastore mounted outside of the ESX host, for administrative purposes, so I decided to check from there. And low and behold this is what I saw.
drwxrwxrwx 17 root root 4096 2010-11-01 14:43 .
drwxr-xr-x 20 root root 4096 2010-11-01 15:21 ..
drwxr-xr-x 2 nobody nogroup 4096 2010-09-20 14:31 md1
drwxr-xr-x 2 nobody nogroup 4096 2010-10-27 11:23 md2
drwxr-xr-x 2 nobody nogroup 4096 2010-09-19 16:21 mich_bi
drwxr-xr-x 2 nobody nogroup 4096 2010-10-14 18:04 oem11
drwxr-xr-x 2 nobody nogroup 4096 2010-09-16 10:20 orabi11
drwxr-xr-x 2 nobody nogroup 4096 2010-10-03 12:09 rh5.5-m2.kickstart
drwxr-xr-x 2 nobody nogroup 4096 2010-09-12 12:36 rh5-dg1
drwxr-xr-x 2 nobody nogroup 4096 2010-10-14 16:53 rh5-dg1_1
drwxr-xr-x 2 nobody nogroup 4096 2010-09-12 15:46 rh5-dg1_2
drwxr-xr-x 2 nobody nogroup 4096 2010-09-13 11:05 rh5-dg2
drwxr-xr-x 2 nobody nogroup 4096 2010-09-12 15:08 rh5-dg2_1
drwxr-xr-x 2 nobody nogroup 4096 2010-11-01 15:07 rh5-rac1
drwxr-xr-x 2 nobody nogroup 4096 2010-11-01 15:09 rh5-rac2
drwxrwxr-x 2 admin admin 4096 2010-11-01 14:43 shared
drwxrwxrwx 10 root root 4096 2010-11-01 12:00 .snapshot
The owner was nobody:nogroup.
I then asked the storage admin to please check the export on the NetApp filer was defined and got this back
/vol/DBM/DBM_QT -sec=sys,rw
I remembered from a while back in my first VCP course that ESX hosts have to have root access to the NFS mount in order to work.
Changed the export to:
/vol/DBM/DBM_QT -sec=sys,rw,root=1.1.2.0/24:1.1.3.0/24
I now tested the creation of a new VM from the vSphere client and verified that the permissions were now created with root:root.
Strangely enough the ESX host was able to mount the datastore, machines were working as well.
I did however still have to change the permissions on the folders and files that were already created on this datastore while the mount was not correct. This I did in order to prevent further problems and rectify the incorrect permissions
chown -R root:root /vmfs/volumes/IT_DBM_QT/*
Of course this was all done with the VM's powered off!