Exchange 2007 CCR is a wonderful technology which provides many new directions in which an Exchange admin can make their environment more resilient and available – and of course by using it you are no longer tied to the “close proximity” model of cluster nodes which previous versions of Exchange clustering required. However, with every new technology there are compromises which every admin must learn to deal with.

One of the common questions that I have been asked (and I have seen a forum posts about) is how do you safely Shutdown or Reboot a CCR cluster node?

Now there are have been a number of solutions that I have seen and read about – many of which are also linked into the question of “How do I fail a CCR cluster over to the passive node”.

Scott Schnoll wrote an excellent article over on the MSExchange Team’s web-site (located here) which described what are the best tools to use when managing CCR clusters (for example is it safe to use the Cluster manager to fail nodes over in a CCR configuration) – however the upshot of this article was:

If you are using CCR Clustered – the Cluster Administrator Tool and Cluster.exe should not be used to manage the Exchange cluster because:

• No validation or health checks are made on the passive copy of the database – if you have issues with the passive copy and the fail-over does not complete you are left in a situation where you down-time is longer while you fix the issues with passive node.
• In certain circumstances using the above tools can leave a database offline because replication is in a broken condition (I have seen this with Public Folders).

In view of the above what are considered the “best practice” when dealing with the situation of moving resources between nodes under CCR? – well that answer to that is to use the Exchange Management Shell move-clusteredMailboxServer cmdlet which does perform such checks – or – if you have installed Exchange 2007 Service Pack1 you can use the “Managed Clustered Mailbox Wizard”.

A very quick run through of the Clustered Mailbox Wizard:

In Service Pack 1 of Exchange 2007 the Exchange team introduced the Clustered Mailbox Wizard into the Exchange Management Console.

It is designed to fill in the gaps that the Cluster Administrator and Cluster.exe tools do not meet, and provide a means where Exchange Admins that might not have gotten to grips with the Exchange management shell commands for Cluster Management can safely manage the cluster.

The following is a quick overview of the CMW in Exchange 2007 SP 1:

Open a connection of a node which is part of your CCR cluster – navigate to [ Start -> Programs -> Microsoft Exchange Server 2007 -> Exchange Management Console ] – when the console loads, expand the “Server Configuration” node and then from the right hand plane Right Click the name of your CCR cluster (see below):

From the context menu that appears select the “Manage Clustered Mailbox Server” option – this will present you with the following dialog box:

You will see that the name of your cluster is preselected and that you are presented with 3 options:

• Move the clustered mailbox server to another node
• Start the clustered mailbox server
• Stop the clustered mailbox server

In this example I will take you through we will be using the “Move the clustered mailbox server to another node” (all of the other examples are very straight forward) – so ensure that the top option is selected and then click on the “Next” button – doing so will change the screen to look like the following:

From the dialog box above click on the button next to the “Target Machine” option and from the box that appears select the node that you wish to move the Exchange resources to, then click on “Ok” – then provide a reason why you would like to move the resources to the new node. When you have filled in the move comment click on the “Next” button which will display a screen which looks like the following:

This is the summary screen of the actions that you wish to take – please review them and when you are happy click on the “Move” button – which will take you to the following screen:

As with many completion screens in the Exchange 2007 you can copy the PowerShell cmdlet used and paste it into Notepad for future reference.

As mentioned before you can use this wizard to perform tasks such as stopping your CCR and Starting your CCR all safe in the knowledge that the correct verification and checks have been made prior to actions being taken.

Right, but what about Shutting down / Rebooting a node?

So, we have very quickly established what to do when you wish to transfer resources to the passive node – but, if the Cluster Administrator and Cluster.exe tools are not aware of Exchange CCR – then what about Windows shutdown or Reboots?

Well the Windows Shutdown processes are also not aware of CCR clustering, and as expected a fail-over will occur when the node is closed down (in some cases this will produce a “lossy” fail-over which can effect Public Folders remounting on the CCR) or rebooted. Consider the following scenario where you are part of a team which has a number of Exchange Admins however some of them are not totally aware of the facts as have been presented above and they decide (as part of some form of routine maintenance) to close down or reboot the primary node using [ Start -> Shutdown -> Shutdown ] – how can you prevent this situation from occurring and mitigate the problems that it might cause?

I have found myself in this situation – as the night shift team where I work does not have a great understanding of clustering and have rudimentary experience in Exchange – however they perform a number of Windows updates per week when required (which can mean either Shutting down or Rebooting the cluster nodes) so I came up with a PowerShell and Group Policy Based Solution where I replace the “Shutdown” button completely on the nodes of my CCR nodes with a new command (Shutdown – CCR Cluster) – see below:

You will notice that the only other command available is “Log off” – we will deal with the configuration of this a little bit later.

From the image above you will see that there is a new option entitled “Shutdown – CCR Cluster” when you click on the options (on either node of the CCR cluster – bearing in mind that this is based around a 2 node CCR cluster) you will be presented with the following screen:

From here you can choose to type “S” to shutdown the node or “R” to reboot the node – for this example I have used “R” for reboot (the entire sequence is the same if you choose “S”. By choosing “R” and pressing <enter> you will be presented with the following screen:

The above screen means that an available node has been found to accommodate the Exchange CCR instance and the script is moving it to the secondary node (actually the script has issued a stop-clusteredMailboxServer and a move-clusteredMailboxServer cmdlet – after performing some WMI analysis)

IF however the secondary node is NOT available or ready to take on the duties of the cluster you will be presented with the following screen:

In this case the node which you have instigated the command from will be either closed down -or– rebooted (depending on the option that was chosen when the Shutdown was selected) – however you can CANCEL the operation by typing in “C”.

If everything is good to go the PowerShell script will stop the clustered mailbox server, then move it to the secondary node and then bring it online.

Then the script will move the Cluster Group over to the Secondary Node and then initiate either a reboot or shutdown of the system (depending on the option that you selected at the beginning of the script.

Ok, I like this how do I implement it?

In order to implement the CCR aware shutdown scripts you will need to create the following Directory on EACH node of your CCR cluster:

C:\PS\

CCR-CLS.ps1 [3KB] – Powershell Batch Script

CCR-Shutdown.bat [1KB] – Windows Batch File

When you have done that download the following two files to the C:\PS\ folder on both your servers.

Open the CCR-CLS.ps1 script using a Text Editor (such as Notepad) – and review the top section – see below;

You will see three values enclosed within a comments section these three values are:

• $Node1 – Primary Node of the CCR Cluster
• $Node2 – Secondary Node of the CCR Cluster
• $Cluster – The network name of the Exchange CCR Cluster

You will see that each of these entries has a “Primary Node Name” or “Secondary Node Name” place holder entry – this must be changed to reflect the names of your cluster nodes and the CCR cluster, for example if my Primary Node was called x64EXCCR1 $Node1 would look like the following:

• $Node1 = “x64EXCCR1”

You will need to complete these changes to the PS1 script on both cluster nodes.

Now that we have modified the scripts to work with the CCR cluster, we now need to remove the normal “Windows” Shutdown command which I accomplished via Group Policy.

I have posted a video below which takes you thought the steps and indeed the settings that need to be changed in GP, however the following is a summary of changes needed:

• Create a new OU in Active Directory that will contain the CCR Cluster Nodes for Exchange
• Move the Cluster Nodes for Exchange into the OU
• Create a new GPO Object and assign it to the new OU, block inheritance and stipulate “no override”
• Edit the GPO object so that the following settings take effect:
  • [ Computer Configuration -> Administrative Templates -> System -> Group Policy ] – User Group Policy loopback processing mode set to enabled and “merge”
  • [ Computer Configuration -> Administrative Templates -> System -> Group Policy ] – Registry policy processing set to enabled and process even if the GPO objects have not changed
  •  [ User Configuration -> Administrative Templates -> Start Menu and Task Bar ] – Remove and Prevent Access to the Shutdown Command set to Enabled
• Change the GPO object settings so that they apply to Domain Admins and Enterprise Admins
• Allow for Domain Replication
• Run GPUPDATE /force on the server and the log off and on again

Below is a detailed video of the above steps you will need either WinRAR or 7-Zip:

ConfigureGPO.rar [434KB – compressed – 32.4MB expanded]

 When the above is configured and you log back onto one of the nodes you will notice that your start menu will look like the following:

Or if you not using the classic view it will look like the following:

And if you Hit CTRL-ALT-DEL you should notice that the “Shutdown” option is grayed out from the Security Dialog box – see below:

The following video shows how you can add the Powershell CCR aware Shutdown Command to the Start menu, however the following is a summary of the actions that need to be taken on BOTH nodes:

• Right click on the Start menu and from the context menu that appears choose the “Open – All Users” option
• Navigate to the C:\PS area and drag a Shortcut to the “CCR-Shutdown.bat” file into the “All Users Start Menu”
• Rename the Shortcut of the batch file in the Start menu to “Shutdown – CCR” and then change its default Icon to that of the Shutdown Command

AddingToStartMenu.rar [287KB – compressed – 12.6MB – expanded]

I hope that you all like the article above, below I have enclosed a view of the source code for the PS Shutdown Script:

# - Change the values below to reflect your CCR Node Names and CCR cluster name$Node1 = ""
$Node2 = "<Secondary Node Name"
$cluster ="" ""

# - End custom values

$ActionQ = Read-Host "Would you like to (R)eboot or (S)hutdown (type R or S)"
$ActionFinal = $ActionQ.ToLower()

$PResultND1 = Get-WmiObject -Class Win32_PingStatus -Filter "Address='$Node1'"
$PResultND2 = Get-WmiObject -Class Win32_PingStatus -Filter "Address='$Node2'"

if (($PResultND1.StatusCode -eq 0) -and ($PResultND2.StatusCode -eq 0)){
Echo "The Servers are present - checking WMI Services Status..."

$ND1StatusIS = Get-WmiObject -ComputerName $Node1 win32_Service -Filter "Name='MSExchangeIS'"
$ND1StatusSA = Get-WmiObject -ComputerName $Node1 win32_Service -Filter "Name='MSExchangeSA'"

$ND2StatusIS = Get-WmiObject -ComputerName $Node2 win32_Service -Filter "Name='MSExchangeIS'"
$ND2StatusSA = Get-WmiObject -ComputerName $Node2 win32_Service -Filter "Name='MSExchangeSA'"

If ($ND1StatusIS.state -eq "Stopped"){
$ND1CCRPassiveReadyIS = "1"
}else{
$ND1CCRPassiveReadyIS = "0"
}

If ($ND1StatusSA.state -eq "Stopped"){
$ND1CCRPassiveReadySA = "1"
}else{
$ND1CCRPassiveReadySA = "0"
}
If ($ND2StatusIS.state -eq "Stopped"){
$ND2CCRPassiveReadyIS = "1"
}else{
$ND2CCRPassiveReadyIS = "0"
}

If ($ND2StatusSA.state -eq "Stopped"){
$ND2CCRPassiveReadySA = "1"
}else{
$ND2CCRPassiveReadySA = "0"
}

if(($ND1CCRPassiveReadyIS -eq "1") -and ($ND1CCRPassiveReadySA -eq "1")){
Echo "Moving Exchange CCR Resources to the Passive Node..."

Stop-ClusteredMailboxServer -identity $Cluster -StopReason "Node Close Down" -confirm:$false
Move-ClusteredMailboxServer -identity $Cluster -TargetMachine $Node1 -MoveComment "Node close Down" -Confirm:$false
Start-ClusteredMailboxServer -identity $Cluster
Echo "Moving Cluster Resources to the Passive Node..."
Cluster.exe group "Cluster Group" /move:$Node1
Echo "Working on Server Action..."
shutdown.exe /m \\$Node2 /$ActionFinal /t 5
}elseif (($ND2CCRPassiveReadyIS -eq "1") -and ($ND2CCRPassiveReadySA -eq "1")){
Echo "Moving Exchange CCR Resources to the Passive Node..."
Stop-ClusteredMailboxServer -identity $Cluster -StopReason "Node Close Down" -confirm:$false
Move-ClusteredMailboxServer -identity $Cluster -TargetMachine $Node2 -MoveComment "Node close Down" -Confirm:$false
Start-ClusteredMailboxServer -identity $Cluster
Echo "Moving Cluster Resources to the Passive Node..."
Cluster.exe group "Cluster Group" /move:$Node2
Echo "Working on Server Action..."
shutdown.exe /m \\$Node1 /$ActionFinal /t 5
}
}else{
Echo "The Server is not present - No nodes appear to be available to OWN the Exchange Instance."
Echo "Do you wish to close down Exchange on this node and then Shut the computer down? (Y or N)"
$ResponseQ = Read-Host "(Y)es or (N)o"
$ResponseFinal = $ResponseQ.ToLower()

if ($ResponseFinal -eq "y"){

Stop-ClusteredMailboxServer -Identity $Cluster -StopReason "Node Shutdown" -confirm:$false
shutdown.exe /$ActionFinal /t 5

}else{

}

}