Configuring SSH Equivalence for Oracle RAC

SSH Equivalence is one of the pre-requisites needed for an Oracle RAC installation.

There are a number of posts on how to do this like here or here, and Oracle even have been so kind as to provide a script that will do this for you (even though it is not 100% automated.
The process is relatively simple (when you break it down piece by piece)

Create the .ssh directory under the users /home folder for VM1 and VM2
Create an RSA key on VM1 and VM2
Copy the contents of ~/.ssh/id_rsa.pub from VM1 and VM2 into ~/.ssh/authorized_keys on both VM1 and VM2
You should then be able to connect to each host (and also the localhost as well) without a password prompt.
Repeat the process on both VM’s with the oracle user

But this process requires a decent amount of manual interaction from the user at the following stages:

Copying the files between VM1 <-> VM2
First connection prompts to add the hosts key to the ~/.ssh/known_hosts file

Manual interaction is the mother of all headaches when you want to automate something. As I have posted before here and here I am in the middle of automating a Oracle RAC deployment on VMware. This is an additional part of the solution.
I had to come up with a method to do this without any user interaction, and here is how I went about the process. I broke down the whole process – stage by stage.

Re-create the ssh_host_rsa_key – the reason for this being – that since these VM’s are deployed from the same template – the ssh_host_rsa_key is identical – and this caused problems for my script (this actually could be useful in some cases – but not here).
Create the ~/.ssh/id_rsa.pub key for the root user on each host – without prompts.
In order to prevent the popup when connecting to another VM for the first time I needed to get the keys from ssh_host_rsa_key.pub into the .ssh/known_hosts before I connected to the VM for the first time.
Add the public key from each VM into the ~/.ssh/authorized_keys file.
Get this information from VM1 to VM2 and and vice-versa – and all of this without prompts – which meant I could not go through the guest operating system.
Repeat the process for the oracle user.

So my initial challenge was how to do the copying of the files without going through the guest OS, but that actually turned out to be pretty simple. PowerCLI has the Copy-VMGuestFile cmdlet that will allow me to transfer files to and from the guest – so that solved my worries.
There were several issues along the way that I needed to address.

I needed to construct the known_hosts file based on several pieces of information, the hostname, IP address and the ssh_host_rsa_key, I am sure there is a easier way of doing this in bash – but this way works for me.
Creating the rsa keys for the oracle user – since I did not want to connect to the VM twice with two different credentials – here I solved the problem by duplicating the files from the root user to the oracle user and manipulated the contents a bit to suit my needs.
Copying the files back to the guest after manipulation – resulted in a change in their format from UNIX to DOS and I could not find a way to control that from the PowerCLI side – therefore some vi manipulation was needed to convert them back.

So without further ado – here is the script – annotations are at the bottom

<#
 .SYNOPSIS
  Configure SSH equivalence between two Oracle RAC nodes

 .DESCRIPTION
  The script will execute on both guests, configure the RSA keys,
  known_hosts and authorized_keys files on each host for both the 
  root and oracle user to enable SSH equivalence for Oracle RAC

 .PARAMETER  VM1
  Name of the first VM
 .PARAMETER  VM2
  Name of the first VM
 .PARAMETER  VM1_IP
  The IP address of the first VM
 .PARAMETER  VM2_IP
  The IP address of the second VM
 .PARAMETER  HostCredentials
  The credentials for the ESXi host
 .PARAMETER  GuestCredentials
  The credentials for the guest VM 
 .PARAMETER Cleanup
  Will cleanup the temporary files created. On by default
 
 .EXAMPLE
  PS C:\> Set-SSHKeys -VM1 hosta -VM2 hostb -VM1_IP 10.10.10.1 -VM2_IP 10.10.10.2
  This example shows how to call the Configure-SSHKeys against hosta with the IP address
  of 10.10.10.1 and hostb with the IP address of 10.10.10.2.
 .EXAMPLE
  PS C:\> Set-SSHKeys -VM1 hosta -VM2 hostb -VM1_IP 10.10.10.1 -VM2_IP 10.10.10.2 -HostCredentials `
  (Get-Credential) -GuestCredentials (Get-Credential) -Cleanup:$false
  This example shows how to call the Configure-SSHKeys against hosta with the IP address
  of 10.10.10.1 and hostb with the IP address of 10.10.10.2. while prompting for credentials
  for both the host and the guest and not cleaning up the files after completion.

 .NOTES
  Author: Maish Saidel-Keesing
  Date: 20 January, 2012
  For more in depth info on the script please see:
  http://technodrone.blogspsot.com/2013/01/set-sshkeys.html

#>
function Set-SSHKeys {
 [CmdletBinding()]
 param(
  [Parameter(Position=0, Mandatory=$true)]
  [System.String]$VM1,
  [Parameter(Position=1, Mandatory=$true)]
  [System.String]$VM2,
  [Parameter(Position=2)]
  [System.String]$VM1_IP,
  [Parameter(Position=3)]
  [System.String]$VM2_IP,
  $HostCredentials,
  $GuestCredentials,
  $Cleanup=$true
 )
 # Check for parameters
 if (!$HostCredentials) {
  $HostCredentials = $Host.ui.PromptForCredential("ESXi Host Credentials","Enter the credentials for the ESXi Host","root","")
 }
 if (!$GuestCredentials) {
  $GuestCredentials = $Host.ui.PromptForCredential("Guest VM Credentials","Enter the credentials for the guest VM","root","")
 }
 if (!$VM1_IP) {
 $VM1_IP = (Get-VMGuestNetworkInterface -Name eth0 -vm $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials).IP
 }
 if (!$VM2_IP) {
 $VM2_IP = (Get-VMGuestNetworkInterface -Name eth0 -vm $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials).IP
 }
## script to be executed on VM1
$myscript1 = @"
mv /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_rsa_key.old
mv /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_dsa_key.old
ssh-keygen -t rsa -N "" -f /etc/ssh/ssh_host_rsa_key
ssh-keygen -t dsa -N "" -f /etc/ssh/ssh_host_dsa_key
mkdir ~/.ssh
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
echo -n `$(hostname -s) >> .ssh/known_hosts
echo -n "," >> .ssh/known_hosts
echo -n $VM1_IP >> .ssh/known_hosts
echo -n " " >> .ssh/known_hosts
cat /etc/ssh/ssh_host_rsa_key.pub >> .ssh/known_hosts
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
mkdir /home/oracle/.ssh
cp .ssh/* /home/oracle/.ssh/
chown -R oracle:dba /home/oracle/.ssh
"@
## script to be executed on VM2
$myscript2 = @"
mv /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_rsa_key.old
mv /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_dsa_key.old
ssh-keygen -t rsa -N "" -f /etc/ssh/ssh_host_rsa_key
ssh-keygen -t dsa -N "" -f /etc/ssh/ssh_host_dsa_key
mkdir ~/.ssh
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
echo -n `$(hostname -s) >> .ssh/known_hosts
echo -n "," >> .ssh/known_hosts
echo -n $VM2_IP >> .ssh/known_hosts
echo -n " " >> .ssh/known_hosts
cat /etc/ssh/ssh_host_rsa_key.pub >> .ssh/known_hosts
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
mkdir /home/oracle/.ssh
cp .ssh/* /home/oracle/.ssh/
chown -R oracle:dba /home/oracle/.ssh
"@
 # run the scripts on VM1 and VM2
 Invoke-VMScript -vm $VM1 -ScriptText $myscript1 -ScriptType bash -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Invoke-VMScript -vm $VM2 -ScriptText $myscript2 -ScriptType bash -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 ## authorized_keys for root
 # get files from guests
 Copy-VMGuestFile -GuestToLocal -Source /root/.ssh/authorized_keys -Destination ./authorized_keys_VM1_root -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -GuestToLocal -Source /root/.ssh/authorized_keys -Destination ./authorized_keys_VM2_root -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -GuestToLocal -Source /home/oracle/.ssh/authorized_keys -Destination ./authorized_keys_VM1_ora -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -GuestToLocal -Source /home/oracle/.ssh/authorized_keys -Destination ./authorized_keys_VM2_ora -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 # Change root to oracle to fix running the script with root credentials
 Get-Item .\authorized_keys_*ora | % {
 (get-content $_).Replace("root@","oracle@") | Set-Content $_ -Force
 }
 # concatenate contents of files
 (Get-Content ./authorized_keys_VM1_root) + "`r`n" + (Get-Content ./authorized_keys_VM2_root) + "`r`n" + (Get-Content ./authorized_keys_VM1_ora) + "`r`n" + (Get-Content ./authorized_keys_VM2_ora) | Out-File -FilePath ./authorized_keys -Encoding ascii
 # copy files back 
 Copy-VMGuestFile -LocalToGuest -Source ./authorized_keys -Destination /root/.ssh/ -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials -Force
 Copy-VMGuestFile -LocalToGuest -Source ./authorized_keys -Destination /root/.ssh/ -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials -Force
 $vicmd = "/bin/vi +':w ++ff=unix' +':q' .ssh/authorized_keys"
 $return1 = Invoke-VMScript -ScriptText $vicmd -vm $VM1,$VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 ## known_hosts for root
 # get files from guests
 Copy-VMGuestFile -GuestToLocal -Source /root/.ssh/known_hosts -Destination ./known_hosts_VM1 -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -GuestToLocal -Source /root/.ssh/known_hosts -Destination ./known_hosts_VM2 -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 # concatenate contents of files
 (Get-Content ./known_hosts_VM1) + "`r`n" + (Get-Content ./known_hosts_VM2) | Out-File -FilePath ./known_hosts -Encoding ascii
 # copy files back 
 Copy-VMGuestFile -LocalToGuest -Source ./known_hosts -Destination /root/.ssh/ -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -LocalToGuest -Source ./known_hosts -Destination /root/.ssh/ -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 $vicmd = "/bin/vi +':w ++ff=unix' +':q' .ssh/known_hosts"
 $return1 = Invoke-VMScript -ScriptText $vicmd -vm $VM1,$VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials 
 ## authorized_keys for oracle
 # copy files back 
 Copy-VMGuestFile -LocalToGuest -Source ./authorized_keys -Destination /home/oracle/.ssh/ -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials -Force
 Copy-VMGuestFile -LocalToGuest -Source ./authorized_keys -Destination /home/oracle/.ssh/ -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials -Force
 $vicmd = "/bin/vi +':w ++ff=unix' +':q' /home/oracle/.ssh/authorized_keys"
 $return1 = Invoke-VMScript -ScriptText $vicmd -vm $VM1,$VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials 
 ## known_hosts for Oracle
 # copy files back 
 Copy-VMGuestFile -LocalToGuest -Source ./known_hosts -Destination /home/oracle/.ssh/ -VM $VM1 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 Copy-VMGuestFile -LocalToGuest -Source ./known_hosts -Destination /home/oracle/.ssh/ -VM $VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials
 $vicmd = "/bin/vi +':w ++ff=unix' +':q' /home/oracle/.ssh/known_hosts"
 $return1 = Invoke-VMScript -ScriptText $vicmd -vm $VM1,$VM2 -HostCredential $HostCredentials -GuestCredential $GuestCredentials 
 # remove temporary files
 if ($Cleanup) {
  Get-Item .\authorized_keys*, .\known_hosts* | Remove-Item -Confirm:$false 
 }
}

Lines 45-57 - The script requires some parameters (two are mandatory). The name of the VM’s that will be configured, their IP addresses, and if you would like to not remove the files created during the process, you should change the $Cleanup variable to $false (by default $true). Also in order to run scripts on the guests you will need to provide credentials for the hosts and the guests (I am assuming that all hosts have one password and also the guests have one password as well).

Lines 59-70 - If the credentials were not provided as variables – then you will be prompted. If the IP’s were not provided, they will be retrieved through the API.

Lines 72-106 - The script that should be run on the guests. There is one for each VM – due to the fact that the IP is (of course) different on each of them.

A bit more details about the script that is run on the guest.

Lines 73-76 - The guest SSH keys are re-created as I explained above.

Lines 77-78 – Create the .ssh directory and create the keys. –N is to set a blank password on the key and –f is for the path.

Lines 79-83 - The known_hosts file is basically a concatenation of 3 things for each entry:
<hostname>, <IP_Address> <Contents of rsa_key.pub> (The commas and spaces are important!)

Line 84 - Add the contents of id_rsa.pub to the authorized_keys file.

Lines 85-87 – Copy the files into the oracle user’s directory and make sure sure the file ownership is correct.

Lines 108-109 – Run the scripts on each VM.

Lines 112-115 – Copy the files to the local computer for text manipulation.

Lines 117-118 – The authorized_keys are per user, and the ones we created for the oracle user were copies of those from the root user, so the username has to be changed.

Line 121 – Combine all 4 authorized_keys files into one, with carriage returns after each one.

Lines 123-126 – Copy the files back to the guests. And as I said above, the files needed some additional vi manipulation because during the copy back – they file type was incorrect.

Lines 129-137 – The same process for the known_hosts file. Take note – only one copy from each guest was needed, that is because it is VM specific and not user specific.The same vi manipulation as well.

Lines 140-149 – The process is repeated to place the files in the oracle user’s home directory.

Lines 151-153 – Cleanup the files – done by default.

2013-01-24

Configuring SSH Equivalence for Oracle RAC