Exchange 2010 introduced a vast amount of changes to the High Availability model with the addition of the Database Availability Group (DAG).  Some features of the DAG are having up to 16 members, automatic database *over to another site as long as you still have quorum, and much more.  Exchange also introduced Database Activation Coordination (DAC) mode as an optional addition to the new High Availability model to prevent split brain syndrome from occurring during a site failover when utilizing a multi-site DAG configuration with at least 3 DAG members and more than one Active Directory Site.  DAC is off by default and in Exchange 2010 RTM it should not be enabled for:

• 2 member DAGs
• Non-Multisite DAGs
• Multi-site DAGs that are in the same stretched Active Directory Site

In Exchange 2010 SP1,  the following changes are introduced and supported for DAC:

• DAGs that contain 2 or more members
• DAGs that are stretched across a single AD Site

Majority Node Set

Before we understand how DAC works, we really have to understand the Cluster Model that DAGs utilize.  Both Exchange 2007 and Exchange 2010 Clusters use Majority Node Set Clustering (MNS).  This means that 50% of your votes (server votes and/or 1 file share witness) need to be up and running.  The proper formula for this is (n / 2) + 1 where n is the number of DAG nodes within the DAG. With DAGs, if you have an odd number of DAG nodes in the same DAG (Cluster), you have an odd number of votes so you don’t have a witness.  If you have an even number of DAGs nodes, you will have a file share witness in case half of your nodes go down, you have a witness who will act as that extra +1 number.

So let’s go through an example.  Let’s say we have 3 servers. This means that we need (number of nodes which is 3 / 2) + 1  which equals 2 as you round down since you can’t have half a server/witness.  This means that at any given time, we need 2 of our nodes to be online which means we can sustain only 1 (either a server or a file share witness) failure in our DAG.  Now let’s say we have 4 servers.  This means that we need (number of nodes which is 4 / 2) + 1 which equals 3.  This means at any given time, we need 3 of our servers/witness to be online which means we can sustain 2 server failures or 1 server failure and 1 witness failure.

Database Activation Coordination (DAC)

In short, DAC mode is enabled when you have at least 3 members to prevent split brain syndrome.  It’s as simple as that. Let’s take a look at an example and see how DAC can help. The longer explanation below talks about this specific model.

Prevention of Split Brain Syndrome

Short Explanation

When the Primary Site goes offline (or we lose too many servers – refer to Majority Node Set above), the Secondary Site will need to be manually activated should you make the choice that a secondary site activation will be required depending on the magnitude of the failure and how long you anticipate the primary site or servers there will be down.  But, when the Primary Site comes back online, the WAN link may be offline.  Because the Primary Site’s Exchange Servers don’t necessarily know about the Manual Site Switchover, they will come up thinking they have Quorum since the Primary Site has the majority of the servers and they are still connected to the old FSW.  Because of this, they will begin to mount databases since to them, they still have Quorum.

DAC mode will enable the usage of a new protocol, Database Activation Coordination Protocol (DACP). This means that DAG members start up with a special memory bit of 0.  They need to contact another DAG node with this special memory bit set to 1.  This memory bit will be set to 1 on one of the DAG members in the Secondary Site since that site is hosting active databases.  Because the WAN link is down, the Primary Site’s DAG members that just came online won’t be able to contact this DAG member with the special memory bit set to 1.  Because of this, they won’t be able to mount databases.  The WAN link will have to come back online which means the Primary Site’s DAG members will now be able to contact the DAG member that has the special memory bit set to 1 which will now allow the Primary Site’s DAG Members to be in a state where they are allowed to mount databases.

Longer Explanation

We can see in this example, there are 5 DAG nodes and no FSW as we have an odd number of DAG nodes.  Our entire Primary Datacenter Fails (or we lose too many servers – in our case, this would be (5 / 2) + 1 which means 3 of our nodes need to remain operational for the DAG to remain operational), the Secondary Site will need to be manually activated should you make the choice that a secondary site activation will be required depending on the magnitude of the failure and how long you anticipate the primary site or servers there will be down.

Part of the failover process will have us shrink the DAG by removing the DAG nodes in the Primary Site from the cluster so all that remain of the existing 2 DAG nodes in the Secondary Site.  Instructions for shrinking the DAG and doing a manual site actiavtion is located here.  Should we decide to proceed with a a manual site switchover , we will provision the FSW in the secondary site during manual site activation to the secondary datacenter.  But what happens if the Primary Site’s Exchange Servers come back online?  They will think they have majority because the primary site has the majority of the servers and the FSW is located there.  Because of this, when they start up, they will begin mounting databases.

Now this is where DAC comes in.  Without DAC enabled, the Primary Site’s Exchange Servers would indeed come online, think they have majority, and begin mounting databases and you run into a split-brain syndrome scenario.  This is because when power is restored to the datacenter, the servers will usually come up before WAN connectivity is fully restored.  The servers cannot communicate with each other between the sites to see that the active databases are already mounted, and because of that, the Primary Exchange Servers will see they have majority since the majority of your servers and your FSW should be in the Primary Site, and mount the databases.

If the servers were allowed to mount databases, and you ran into a split-brain scenario, something called Database Divergence would occur. Database Divergence is where the databases in the primary site would become different from the secondary site causing  the need for a reseed from the authority database which would cause some database loss from the new database that went into the diverged database due to split-brain from occurring.

The way DAC works, is that all servers have a new protocol known as Database Activation Coordination Protocol (DACP).  One of the DAG Nodes will always have a special memory bit set to 1. What this means is, with DAC on, any time a server wants to mount a database, it will have to find a server with the DAC memory bit set to 1.  Because of this, when the Primary DAG Servers come back online, they will need to contact a server with that memory bit set to 1 in order to be in a state where it can begin mounting databases.  Because the WAN is down, these Primary Datacenter DAG Servers that are now just coming back online won’t be able to mount databases because none of these servers will have that special memory bit set to 1.  That memory bit will be set on one of the DAG Servers in the Secondary Site. Once WAN connectivity is restored, these Primary Datacenter DAG Servers will now be able to communicate with the DAG Server that happens to have that special memory bit set to 1 and now these DAG Servers will be allowed to mount databases.

Thankfully, in SP1, DAC will work with  2 node DAGs and multi-site DAGs that are using a stretched AD Site.

DAC and ForceQuorum

If you do not know what Forcequorum is,  have a quick look at my blog post here. Essentially, forcequorum allows you to forcefully start a cluster when this cluster has lost quorum.  You’re forcing it to bypass the Majority Node Set requirement to become operational.  In CCR, forcequorum was used in a geographically dispersed CCR cluster.  When the Primary Site went offline, you had to run forcequorum on the node in the Secondary Site and then set a new File Share Witness.  This is similar in Exchange 2010 DAGs when the Primary Site goes offline.

The article here is entitled Datacenter Switchovers and is the article to use when planning Site Resiliency with Exchange 2010.  You can see, in the procedure for terminating a failed site, there are two methods:

• When the DAG is in DAC mode:
• When the DAG isn’t in DAC mode

When looking at the procedures for when DAC is NOT enabled, there are more steps that have to be done which involve running clussvc commands.  When looking at the procedures for when DAC is enabled, there are no steps which involve running clussv commands.  This is because when you have DAC mode on, DAC takes care of all the clussvc procedures for you in the background. As you can see, it is well worth it to ensure you have at least 3 DAG nodes in a DAG just to utilize DAC.  But again, in Exchange 2010 SP1, DAC can be utilized with DAGs that contain two nodes.

Simple Script Features (CSV Download)

1. $DefaultPIN is exactly that.  The CSV has a PIN column which is empty by default.  If this CSV field is left blank for a given user, it will use the $DefaultPIN.  Otherwise, it will use the PIN specified in the script.
2. The Script will search for non-legacy Mailboxes (non-Exchange 2000/2003 Mailboxes if running Exchange 2007) and use the First and Last column (for the user’s first name and last name)  in Excel.  Because Firstname and Lastname is not unique in AD, the script will error on a user if they have multiple mailboxes.  It will tell you to enter the user’s Alias in that user’s Alias column in Excel.  The script will then get the mailbox that has that Alias.  This doesn’t just rely on Get-Mailbox -identity alias because that can still return multiple mailboxes.  It does a Get-Mailbox -identity alias but also does a Where-Object {$_.alias -eq $Mailbox.Alias} to ensure we use the correct mailbox.
3. The script will enable the user’s UM Mailbox based on the Mailbox GUID on the mailbox that is retrieved to ensure enable the correct Mailbox based on the unique (GUID) identifier.
4. Allows you to set the personal operator extension of a user based on the information in the CSV.  If you don’t need to set the Operator Extension, just go into the script and remove the Set-UMMailbox line as everything else is contained in the Enable-UMMailbox line.
5. The script assumes you have one UM Mailbox Policy and uses that to assign users to.

Complex Script Features (CSV Download)

1. Includes all the features of the Simple Script plus the following:
2. Doesn’t send the default SMTP Message to user’s when they are UM Enabled.  The variable $NotifyEmail is where you want the welcome message sent which should obviously be a mailbox you perhaps create for the purpose of sending welcome messages to.
3. Instead of sending the welcome message to the user’s mailbox when they are UM Enabled, the variable $smtpFrom is where you want a custom html formatted welcome message sent from.  This could be something such as ExchangeUMWelcome@domain.com.
4. To tweak the custom HTML Formatted message, go down to the variable $EmailBody and include your own HTML.  By default, it will sent the user their PIN (if $DefaultPIN is used, that is sent to the user and if there’s a PIN for that user in the CSV, it uses that instead), their Extension, and their Subscriber Access Number as defined in the CSV.
5. The script will allow you to choose from two separate UM Mailbox Policies.  By default, the script uses North America and London.  If you have different UM Mailbox Policies which you most likely will, you will need to go down to the Enable-UMMailbox command to tweak the name of the Policies that are used.  If you want to add more, you will need to modify the Write-Host lines near the beginning of the script which gives the user the option what to select and then go down to the Enable-UMMailbox and tweak the elseif pieces to take into consideration the additional UM Mailbox Policies to consider.

Note: The CSV file used for the Complex Script is available for download from here.  The one difference between this and the simple version of the CSV is this CSV contains a SubscriberAccess column which the HTML message captures and uses as a variable to send to the user’s primary SMTP address when enabl

Simple Script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

###############################
# UM Simple Automation v1
# By Elan Shudnow
###############################
 
########## MODIFIABLE OPTIONS ##########
# Set This PIN to the Default PIN.  If the CSV PIN Field is blank, it uses this.  If not blank, it uses the CSV PIN.
$DefaultPIN = 12345
 
# Set this to the location of the CSV file.
$mailboxes =  Import-CSV "UMsimple.csv"
 
########## DO NOT MODIFY ANYTHING BELOW THIS LINE ##########
 
# Call the Loop to Enable Users
Foreach ($mailbox in $mailboxes) {
 
	# By default, Excel will have empty Alias Column so it will search by First Last.
	if (!($mailbox.alias)) {
		$User = Get-Mailbox -Identity ($Mailbox.First + "" + " " + $Mailbox.Last) -ResultSize Unlimited -ErrorAction SilentlyContinue | Where-Object { $_.RecipientTypeDetails -eq "UserMailbox" }
	}
	else {
		$User = Get-Mailbox -Identity $Mailbox.Alias -ResultSize Unlimited -ErrorAction SilentlyContinue | Where-Object { $_.RecipientTypeDetails -eq "UserMailbox" -and $_.alias -eq $Mailbox.Alias }
	}
 
	# By default, Excel will have empty Alias Column so it will search by First Last.
	# This will notify you to modify the Alias Column so that you can search on a unique field if there are multiple
	# mailboxes with the same First Lastname that are on Exchange 2007.  The script ignores Exchange Legacy Mailboxes (Exchange 2000 and Exchange 2003).
	if ($User) {
		if ($User.Count -gt 1) {
			Write-Warning $User "There are multiple users with this First Name and Last Name.  Go into the spreadsheet and provide the alias for the correct mailbox user"
		}
		else {
			if ($User.UMEnabled -eq $false) {
				Enable-UMMailbox -Identity $User.GUID.toString() -ummailboxpolicy $((Get-UMMailboxPolicy).Identity) -pin $(if (!($mailbox.pin)) { $DefaultPIN } else { $Mailbox.PIN }) -pinexpired $true -Extensions $Mailbox.Extension -NotifyEmail $NotifyEmail
				Set-UMMailbox -Identity $User.GUID.toString() -OperatorNumber $Mailbox.Operator
			}
			else {
				Write-Host $User "is already enabled"
			}
		}
	}
	else {
		Write-Host "ERROR:" ($Mailbox.First + "" + " " + $Mailbox.Last) "'s Mailbox Does Not Exist"
	}
}

Complex Script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

###############################
# UM Complex Automation v1
# By Elan Shudnow
###############################
 
########## MODIFIABLE OPTIONS ##########
# Set This PIN to the Default PIN.  If the CSV PIN Field is blank, it uses this.  If not blank, it uses the CSV PIN.
$DefaultPIN = 12345
 
# Set this to the location of the CSV file.
$mailboxes =  Import-CSV "UMcomplex.csv"
 
# Set this to the Notify Email you want.
$NotifyEmail = "notifyname@domain.com"
 
# Set this to the e-mail address where users will receive Welcome Messages From
$smtpFrom = “smtpfrom@domain.com"
 
$smtpServer = "hubserver.domain.com"
 
########## DO NOT MODIFY ANYTHING BELOW THIS LINE ##########
 
# Allows the user running the script to choose which UM Mailbox Policy the group of users in the CSV
# should belong to.  This will assign the policy to all users in the given CSV file.
write-host
write-host Exchange Server 2010 - Unified Messaging Enabling
write-host Please, select which UM Mailbox Policy you want assigned
write-host
write-host '1) North America'
write-host '2) London'
write-host
$location = Read-Host "Select an option.. [1-2]? "
 
function Send-Email {
	Param ($To, $From, $Subject, $Body)
 
	$msg = New-Object Net.Mail.MailMessage
	$msg.From = $From
 
	$msg.To.Add($To)
 
	$msg.IsBodyHtml = $true
	$msg.Body = $Body
	$msg.Subject = $Subject
 
    $client = New-Object net.Mail.SmtpClient($smtpServer)
    $client.Send($msg)
}
 
# Call the Loop to Enable Users
Foreach ($mailbox in $mailboxes) {
 
	# By default, Excel will have empty Alias Column so it will search by First Last.
	if (!($mailbox.alias)) {
		$User = Get-Mailbox -Identity ($Mailbox.First + "" + " " + $Mailbox.Last) -ResultSize Unlimited -ErrorAction SilentlyContinue | Where-Object { $_.RecipientTypeDetails -eq "UserMailbox" }
	}
	else {
		$User = Get-Mailbox -Identity $Mailbox.Alias -ResultSize Unlimited -ErrorAction SilentlyContinue | Where-Object { $_.RecipientTypeDetails -eq "UserMailbox" -and $_.alias -eq $Mailbox.Alias }
	}
 
	# By default, Excel will have empty Alias Column so it will search by First Last.
	# This will notify you to modify the Alias Column so that you can search on a unique field if there are multiple
	# mailboxes with the same First Lastname that are on Exchange 2007.  The script ignores Exchange Legacy Mailboxes (Exchange 2000 and Exchange 2003).
	if ($User) {
		if ($User.Count -gt 1) {
			Write-Warning $User "There are multiple users with this First Name and Last Name.  Go into the spreadsheet and provide the alias for the correct mailbox user"
		}
		else {
			if ($User.UMEnabled -eq $false) {
				Enable-UMMailbox -Identity $User.GUID.toString() -ummailboxpolicy $(if ($location -eq 1) { "North America" } else { "London" }) -pin $(if (!($mailbox.pin)) { $DefaultPIN } else { $Mailbox.PIN }) -pinexpired $true -Extensions $Mailbox.Extension -NotifyEmail $NotifyEmail
				Set-UMMailbox -Identity $User.GUID.toString() -OperatorNumber $Mailbox.Operator
				$Extension = $Mailbox.Extension
				$Pin = $(if (!($mailbox.pin)) { $DefaultPIN } else { $Mailbox.PIN })
				$SubscriberNumber = $Mailbox.SubscriberNumber
$EmailBody = @"
Welcome to Exchange Unified Messaging!
 
Your Extension is $Extension
 
Your PIN is $Pin
 
Your Subcriber Access Number is $SubscriberNumber
 
"@
 
				$EmailSub = “Welcome to Exchange Unified Messaging!”
				$EmailTo = $User.PrimarySmtpAddress
				$EmailFrom = $smtpFrom
				Send-Email $EmailTo $EmailFrom $EmailSub $EmailBody
			}
			else {
				Write-Host $User "is already enabled"
			}
		}
	}
	else {
		Write-Host "ERROR:" ($Mailbox.First + "" + " " + $Mailbox.Last) "'s Mailbox Does Not Exist"
	}
}

There’s a bug with Outlook 2007 and Outlook 2010 that prevent the RPC Endpoint from updating.  This is similar to the bug in which Outlook 2007 will not update its Outlook Anywhere Endpoint which was fixed in Outlook 2010 Beta 2.  Because of this bug, it is very important that you get the RPCClientAccessServer database parameter configured correctly prior to moving users to Exchange 2010.  If you make this mistake and have the RPCClientAccessServer Database Parameter incorrectly and Outlook users are already hosted on Exchange 2010, once you modify the RPCClientAccessServer parameter for those clients, those clients can do an Outlook Profile Repair to get the updated change.  I will update this article in the future when this issue gets fixed.

So before you move users to Exchange 2010, please make sure that you either:

1. Create the RPC Client Access Array before creating your databases OR
2. Go back on the databases and stamp those databases by running the following command:
Set-MailboxDatabase -Identity “DatabaseName” –RPCClientAccessServer array.domain.com

The reason why #1 works making #2 not necessary is the way Exchange assigns the RPC Client Access Array property to a database.  It does this in 3 different ways:

1. If there is no RPC Client Access Array and you create the database on a server that hosts the MBX and CAS role, it will choose itself, always.
2. If there is no RPC Client Access Array and you create the database on a server where the CAS and MBX are not collocated on the same server, it will randomly pick a CAS in the same site to set as the RPCClientAccessServer.
3. If there is an RPC Client Access Array in that site, it will automatically set the FQDN of the CAS Array for the RPCClientAccessServer.

Autodiscover will see the database the user lives on and will assign the user’s Exchange Server (RPC Endpoint) to what the RPCClientAccessServer parameter is.  That is why it’s important to make sure this setting is right before a user is moved to Exchange 2010.

I won’t go into MRM 1.0 a whole lot but will show you the capabilities of Retention Policies in Exchange 2010, how it ties in with Outlook 2010, and how you can create Retention Policies via the Exchange Management Console in Exchange 2010 SP1.

Important: Please keep in mind that the screenshots below are not final as this article is based off of SP1 Beta software and may change by the time Exchange 2010 SP1 ships.

Retention Policy functionality

Default Policy Tag

A Default Policy Tag is the default Tag that is chosen for a Exchange specific folder such as Inbox, Calendar, Deleted Items, etc.  It affects all subfolders and subitems within that folder.  It is essentially the default policy/tag assigned to a specific folder. An example of a Default Policy Tag is when you enable a Personal Archive Mailbox for a user they are assigned a Default Policy Tag which says that all mailbox data will be moved to the archive after 2 years and it applies to all folders within an Exchange Mailbox.  Alternatively, you can also specify a specific part of the mailbox to be used for the Default Policy Tag instead (such as your Inbox, or Calendar, etc.)

Policy Tags

Policy Tags are the options available for a user to select on a Personal folder which are essentially any user created items which include a subfolder off of the Inbox, items created on your calendar, a user created task, and so on. This Policy Tag which is assigned to a Personal Folder is there to allow a user to override the Default Policy Tag.  It is essentially a way for a user to override the Default Policy (Default Policy Tag) set on a folder or a specific item or subfolder under the main folder that the Default Policy Tag is assigned to. So let’s say we enabled a Personal Archive Mailbox for a user and they have a Default Policy Tag for All Folders in a Mailbox for 2 years to be  pushed down to their Personal Archive Mailbox.  This Default Archive Policy also has a few Policy Tags that allow a user to select a folder or a specific item and choose the following options:

• 1 year
• 5 years
• Never

By selecting 1 year, we are using the Retention Policy Tag that allows the user to select 1 year.  If we select User Folder Policy, we are essentially reverting back to the default setting which let’s the Default Policy Tag (2 years) to govern when that specific item gets moved to the archive.  So think of the Default Policy Tag as the default permission whereas a Policy Tag is a way for a user to override the Default Permission (the Default Policy Tag).

Retention Policy

A Retention Policy contains a Default Policy Tag and Policy Tags.  You assign this Retention Policy to a user.

A Walkthrough of Creating a Retention Policy in the Exchange Management Console

Microsoft is moving away from MRM 1.0.  In fact, in the Exchange 2010 SP1 Exchange Management Console, Managed Folders and Custom Managed Folders which were MRM 1.0 functionality has been removed.  The Exchange Management Shell still has MRM 1.0 functionality.  As you can see by the following image, there’s no Managed Folders or Custom Managed Folder tabs.  But, there is a Retention Policy Tags and a Retention Policies Tab.

The Scenario

Let’s say we have a MRM 1.0 policy that does the following: When an item is moved to Deleted Items, after 14 days it will be deleted with the ability to recover that deleted item.  In MRM 1.0 we couldn’t create any user configurable options.  But with Exchange 2010, we can use Policy Tags to allow our users to override the default we give them. We want to create a similar policy in Exchange 2010 SP1 (RTM did have Retention Policies but not the ability to create them in the Exchange Management Console).  We also want to take advantage of the Default Policy Tag to take care of the 14 day deletion but also provide additional Policy Tags to allow users to be able to choose a different time limit such as 7 days and 21 days.  Let’s go ahead and create a Default Policy Tag to Delete Items after 14 days.

So to recap the goals of our Retention Policy:

• Default Policy Tag to delete all items in the Deleted Items folder after 14 days
• Policy Tag to allow users to override the Default Policy Tag and be able to select 7 days for individual folders and/or items
• Policy Tag to allow users to override the Default Policy Tag and be able to select 21 days for individual folders and/or items

So let’s create our first Retention Policy Tag which will become our Default Policy Tag because we are assigning it to a folder that Exchange creates, the Deleted Items folder.

We then want to allow the user the ability to select 7 days and 21 days. We will assign the Tag Type to Personal Folders which essentially makes it into a Policy Tag rather than a Default Policy Tag and will allow the users in Outlook 2010 to select 7 days or 21 days to override the Folder Policy (Default Policy Tag).

Let’s go ahead and create our 7 day Retention Tag.

Let’s go ahead and create our 21 day Retention Tag.

We can now see the Retention Tags and the Default Policy Tag.

So let’s go over to the Retention Policies Tab and create a new Retention Policy that includes all three of our Retention Policy Tags.

During the Policy creation we can specify mailboxes to associate the policy to; which I did.  After assigning the policy and running the Managed Folder Assistant (Start-ManagedFolderAssistant) to expedite the process of assigning the policy to the mailbox, I launched Outlook 2010 with this user’s account profile.

If we take a look at the Inbox Policy, it just says to use the Parent Policy.

But if we look at the Deleted Items Folder in which we created the Default Policy Tag for, we can see our Default Policy Tag at work.

In order to move the personal archive mailbox, we would need to ensure both the user mailbox as well as the archive mailbox are on SP1.  So if you are in a multi-server environment where some mailbox servers are RTM and SP1 and you have moved the user to SP1 and want to then split off the personal archive mailbox to a separate database, make sure that server that you move the personal archive mailbox to contains SP1. Should you decide to move the mailbox back to RTM, you must move both the user mailbox and the personal archive both to an RTM Server.

So let’s go through two scenarios.  The first, we will move an existing user’s personal archive to a separate database.  The second scenario we will create a new user with a personal archive and have the personal archive live on a separate database.

Important: Please keep in mind that the screenshots below SP1 Beta softwareand even the statement about Microsoft Support are not final as this article is based off of SP1 Beta software and may change by the time Exchange 2010 SP1 ships.

Scenario 1 – Moving an existing Personal Archive Mailbox to a Separate Database

So let’s take a look at a test mailbox I created, Frodo Baggins. We can see that both the user mailbox as well as the personal archive mailbox live on the same database.

So let’s move the personal archive to a separate database, MDB2.

To check the status of the Move Request, we can run the following command:

Now that we see the move has been Completed, we can check again to see what mailbox our user mailbox and our personal archive mailbox are located on. As you will see, our user mailbox and our personal archive will be housed on separate databases.

Let’s say we want to then move the Personal Archive Mailbox to MDB1.  But this time, let’s use the Exchange Management Console. We can see the existing database that both the User Mailbox and the Personal Archive Mailbox lives on.  We can also choose to move only the Personal Archive Mailbox.  We will choose MDB1.

We will then go ahead and verify again that the personal archive has successfully been moved to MDB1.

Scenario 2 – Creating a new User and separating their user mailbox and personal archive mailbox to separate databases

Microsoft has also updated the GUI so that when you create a new user, you can specify what database the user mailbox will live on as well as what database the personal archive mailbox will live on.  Let’s go ahead and create a new Active Directory user and assign this user an Exchange Mailbox as well as a Personal Archive Mailbox.

We will specify the User Mailbox to live on MDB3.

We will specify the Personal Archive Mailbox to live on MDB2 which is different from the User Mailbox which will live on MDB3.

We can then verify in the Exchange Management Shell to see if our User Mailbox is properly in MDB3 while our Personal Archive Mailbox is in MDB2.

A note about Microsoft Support

One important thing to note, is that in order to have support for personal archive mailboxes being on separate databases, both mailbox databases must be located within the same Active Directory Site.  The only time a personal archive will be supported on a database in a separate Active Directory Site is during a failover scenario where the database copy fails and activates on a separate server located in a separate Active Directory site.  But for normal operations, the user mailbox and personal archive mailbox database should be in the same Active Directory Site.

Exchange Load Balancing Options

In Exchange 2007, if you wanted any type of HA, you needed at least four servers.  2 for CCR Nodes and 2 for HUB/CAS Nodes.  The reason why you cannot have 2 nodes altogether is that CCR Nodes were limited to the Mailbox role only.  For an Exchange Site, you need to always have at least the  HUB/CAS/MBX Role  for that site to be operational.  In Exchange 2010, more options are now available.  You now have something called Database Availability Groups (DAGs).  These DAG members can contain all Exchange roles (HUB/CAS/MBX/UM) but still may not contain the Edge Transport role.

There is a problem though.  There is a Windows limitation that allows you to install Windows Network Load Balancing on a server that also contains Failover Clustering Services. So while we can now have 2 Exchange 2010 Servers, we need a way to load balance the CAS role to provide High Availability for the following CAS Services:

• Outlook Web App (formerly Outlook Web Access) (HTTP Traffic)
• Exchange Control Panel (HTTP Traffic)
• Exchange Web Services (HTTP Traffic)
• Exchange ActiveSync (HTTP Traffic)
• Autodiscover (HTTP Traffic)
• Offline Address Book (HTTP Traffic)
• Outlook Anywhere (HTTP Traffic)
• RPC Client Access (RPC  Traffic)

There are a few options for load balancing.  The first is the ability to use ISA.  The problem here, is that ISA can only load balance HTTP-based traffic.  If you take a look at the bulleted list above, you can see that RPC Client Access Service is RPC Traffic which means that ISA cannot load balance this traffic.  We have a few load balancing options then:

1. 2 Multi-Role DAG Members and Hardware Load Balancers – Utilize 2 Multi-Role DAG Members (MBX/HUB/CAS).  Use a hardware load balancer to load balance all of the bulleted items above including the RPC Client Access Service using an RPC Client Access Array which load balances port 135 for the RPC Endpoint Mapper and 1024-65535 ports.  Typically, since you are using High Availability, this means that you would most likely want to have 2 hardware load balancers.



2. 2 DAG Members, 2 HUB/CAS Servers, and Windows Network Load Balancing - Utilize 2 DAG Members (MBX).  Use 2 HUB/CAS Servers with Windows Network Load Balancing.  Windows Network Load Balancing will load balance all of the bulleted items above including the RPC Client Access Service using an RPC Client Access Array which load balances port 135 for the RPC Endpoint Mapper and 1024-65535 ports.



3. 2 DAG Members and DNS Round Robin - Use 2 Multi-Role DAG Members (MBX/HUB/CAS).  Use DNS Round Robin to achieve a “poor man’s solution” type of load balancing.  With this scenario, you will not have automatic failover for the RPC Client Access Service.  You will essentially create two A Record for the RPC Client Access Array; one pointing to the first multi-role DAG Member and one pointing to the second multi-role DAG Member.  You will most likely want to lower the TTL values of these DNS records to 5 minutes so if a failure does happen, you can remove one of the A records and the clients will flush their DNS cache within 5 minutes time.



4. 2 DAG Members, ISA/TMG/UAG, and either Hardware Load Balancing or DNS Round Robin - Use 2 Multi-Role DAG Members (MBX/HUB/CAS).  Use ISA/TMG/UAB to load balance all HTTP items from the bulleted list above. The issue here is that now with Exchange 2010, for mailbox access, users connect to the Client Access Server for their RPC Endpoint.  To make this redundant, we create an RPC Client Access Array.  This RPC Client Access Array can be load balanced through a hardware load balancer, DNS Round Robin, or Windows Network Load Balancing.  ISA/TMG/UAG cannot load balance non-HTTP Traffic.  So if you have ISA/TMG/UAG, you can still use it to load balance all HTTP Traffic but you would still need to use a Hardware Load Balancer, DNS Round Robin, or Windows Network Load Balance to load balance the RPC Client Access Array.  The example picture below will show the use of UAG with a Hardware Load Balance mix.

Exchange Load Balancing Options and their benefits

Taking a look at the above list of options, we can use several different options including Windows Network Load Balancing, Hardware Load Balancing, and DNS Round Robin. Each has their pros and cons in terms of cost and functionality.

Hardware Load Balancing

Hardware Load Balancers can have the most capacity in terms of user connections.  But for SMBs, you won’t have to worry about load.  The load is more for very large organizations.  In fact, Microsoft recommends that if you are going to require over 7 HUB/CAS Servers in a load balanced farm, to use Hardware Load Balancers instead of Windows Network Load Balancing.  Hardware Load Balancers are also the most expensive option.

Hardware Load Balancers do have the best functionality from a perspective of Client to Server Affinity depending on the vendor used.  For example, we can use multiple affinities and have fallbacks to a specific affinity of our preferred affinity fails.  For example, we can set up up our hardware load balancers to use the following affinity in terms of preference:

• Existing Browser-Based Cookie
• Hardware Load Balanced created cookie
• SSL Session ID
• Source IP

The goal here is to make sure that every user is load balanced evenly and that automatic failover can occur quickly and smoothly.

Windows Network Load Balancers

Windows Network Load Balancers do not achieve as much capacity in terms of user connections as a Hardware Load Balancer, but they can still handle a lot of connections.  Windows Network Load Balanced farms can use as many as 8 CAS Servers without suffering a performance degradation.  In order to have the need for 8+ CAS Servers, you’ll need to have many users (tens of thousands). Windows Network Load Balancing is built into Windows Server and therefore, it’s a large cost savings in comparison to purchasing hardware load balancers.

Windows Network Load Balancers do not have as good of functionality of Hardware Load Balancers from a perspective of Client to Server Affinity.  For example, we only have one affinity method.  That method is Source IP.  The downside to using Source IP is if you have a lot of connections coming from a NAT’d Source IP. This means that all of these connections will end up hitting the same Client Access Server as again, the only Affinity Method a Windows Network Load Balancer has is Source IP.

Most likely, if you don’t have the need for more than 8 CAS Servers, Windows Network Load Balancing will suffice for you needs.  It’s cheap, comes with Windows, and does its job.

ISA Server, TMG, or UAG

ISA/TMG/UAG Servers to have more capabilities than Windows Network Load Balancers.  The one downside to them is that they cannot load balance RPC Traffic.  Because of that, you can still use ISA/TMG/UAG to load balance your HTTP traffic, but you’ll still need a Hardware Load Balancer or a Windows Network Load Balancer to load balance your RPC Client Access Array.

ISA/TMG/UAG do scale better than Windows Network Load Balancing but not as well as a Hardware Load Balancer.  ISA/TMG/UAG does not cost as much as a Hardware Load Balancer but is more expensive than Windows Network Load Balancing.  ISA/TMG/UAG also has the capability to do Load Balanced created cookies as well as Source IP Affinity depending on the protocol ISA/TMG/UAG is publishing.

Another upside to using ISA/TMG/UAG is that they can do pre-authentication.  This means that if a server goes down in which a client has affinity to, ISA/TMG/UAG still contains the authentication context of the user and automatically re-authenticates to the new Client Access Server.

DNS Round Robin

DNS Round Robin scales just as high as Hardware Load Balancers because the connections will just go directly to the Client Access Servers.  If anything, it has the highest scale as you don’t have anything in the middle doing anything with the connections.  It’s also free to use!  But in this case, free is not necessarily good because you lose a lot of functionality.  Hardware Load Balancers, Windows Network Load Balancers, and ISA/TMG/UAG all have the capability to detect server failures and automatically stop sending to the server and direct all traffic to a server that is operational.

DNS Round Robin has no automatic server failure detection.  If a host goes down, an Administrator will need to realize it, remove the DNS A/HOST Record for the server that went down, and then clients will have to wait for the TTL value on the old DNS record to expire.  When that happens, the client will begin connecting to the proper server. So you save a lot of money going with this option, but you lose all automation and gain downtime instead.

Scenario #1 – Full Site Failure

Let’s say you have a Client Access Server Array called array.domain.com.  Primary Site goes down.  As a part of the manual site switchover process, you must update the DNS records in your Primary Site to point to the CAS infrastructure at your DR Site.  One out of several DNS records you change will include the CAS Array. You change array.domain.com to point to DRSiteCAS instead of PrimarySiteCAS.  The client (after the DNS record flushes – recommended for TTL value to be 5 minutes for DNS records in site resilient solutions) will then start to connect to the DRSiteCAS which will then access the database in the DR Site.

Scenario #2 – Server Failure(s) in Primary Site and Disabling Automatic Activation for Databases and Servers

In the case where all database copies go down in the Primary Site, your databases can automatically failover to the DR Site as long as you allow automatic activation on the DR Servers (yes, you can turn off automatic activation on databases and servers) and as long as you still have Majority for your Quorum. In this scenario, the RPC Client Access (and array) can access the mailbox databases that are mounted in the DR Site.

Automatic Activation

As I just eluded to above, it is possible to turn off automatic activation on databases and servers. There is something called Database Activation Policy.  Let’s say you wanted to disable a specific database from being considered in the Automatic Activation Process.

You can use the following command to prevent the database from being considered in the Automatic Activation Process:

This example resumes the copy of the database DB1 on the server MBX2 for automatic activation:

This is also possible to do at the mailbox server level using the Set-MailboxServer cmdlet.  You can use the following command to prevent any databases on a specific mailbox server from being considered in the Automatic Activation Process:

This example resumes all database copies on the mailbox server “MailboxServer” for automatic activation:

Example

Let’s say we have 6 DAG Servers with 4 in the Primary Site and 2 in the DR Site with no modifications to the Automatic Activation Policy (DAG Servers in the DR Site can automatically mount databases).  Let’s say, we currently have a lack of funds for storage which prohibit the ability to have mailbox database copies on all servers.  So PrimarySiteMBX01 and PrimarySiteMBX02 in the Primary Site are mirrored in terms of mailbox database copies.  PrimarySiteMBX03 and PrimarySiteMBX04 in the Primary Site are mirrored in terms of database copies.  PrimarySiteMBX01 and PrimarySiteMBX02 are mirrored with SecondarySitMBX0102 in the DR Site and PrimarySiteMBX03 and PrimarySiteMBX04 are mirrored with SecondarySiteMBX0304 in the DR Site.

To make it a bit more clear, the following image shows database distribution.  You can see there are 6 nodes and 3 copies of each database.

Should PrimarySiteMBX01 and PrimarySiteMBX02 go down (as illustrated below), SecondarySiteMBX0102 can automatically mount the database because majority is still there for quorum.  In this case, the RPC Client Access Array in the Primary Site will still successfully be able to provide mailbox access to the databases mounted on SecondarySiteMBX0102 in the DR Site.  This is one of the nice things I like about Exchange 2010 High Availability, is that if your DAGs go down, you can allow the copy in the DR Site to automatically activate (provided the Database Activation Policy as described above allows it to automatically mount) whereas in Exchange 2007, you had to manually activate any SCR copy.

Exchange 2007 and Exchange 2010 Clusters both use Majority Node Set Clustering.  This means that 50% of your votes (server votes and/or 1 file share witness) need to be up and running.  With DAGs, if you have an odd number of DAG nodes in the same DAG (Cluster), you have an odd number of votes so you don’t have a witness.  If you have an even number of DAGs nodes, you will have a file share witness in case half of your nodes go down, you have a witness who will act as that extra +1 number.

So in this scenario, we have 6 votes from the servers plus 1 witness from the file share witness totaling 7 votes.  This means we can have up to 3 servers fail and our cluster will still be online.  This is because if you are in the scenario where we 7 votes, if 3 go down that leaves us with 4 votes which satisfies the 50% + 1 majority rule. Because of this, we still have majority and our quorum and cluster are still fully operational.

Now when exactly would we have to do a manual switchover?  Well, there’s a couple cases.  The first would be if your Primary Datacenter has a complete outage.  This may be due to power failure, environmental disaster, etc…  The other is because all Primary Datacenter DAG members go down or just enough servers go down (again, 50% + 1 voters must be up which means if we lose more than 3 machines (includes FSW), our entire cluster goes offline.  In this case, you’ll have to do a manual datacenter switchover.  You’ll move over all services to the secondary datacenter including changing the RPC Client Access Server FQDN to point to the single CAS Server or the standby VIP that publishes RPC across multiple Secondary Datacenter CAS Servers.

Some blog articles that talk about this include:

http://www.tincupsandstring.com/2009/12/01/forcing-address-book-download/

http://www.markc.me.uk/MarkC/Blog/Entries/2009/12/17_Force_Downloading_the_Address_Book_in_OCS.html

Now I’m sure you are asking yourself why I am creating this entry?  Is it just to repeat information that’s already out there?  Of course not!

So, Communicator 2007 R2 is a 32-bit (x86) application.  That registry entry works perfectly fine on x86 systems.  But, if you are running on a x64 system, it won’t.  Why?  Well, because when you run x86 applications on a x64 based system, it utilizes a system in Windows called Windows on Windows (WOW64).  WOW64 has its own section within the registry called Wow6432Node.

So let’s say we take the registry key for our Communicator x86 (Communicator x64 not available) and run it on an x86 system.  The following registry key works fine:

But let’s say we have an x64 system.  The above registry key will not work.  We need to utilize the WOW6432Node part of the registry.  The following registry key works for x64 systems:

Please make sure you back up your registry before making changes as making changes to the registry can be harmful to your system if not done properly.

The Issue

For both the Distribution Group Mail Flow Message Moderation as well as the Membership Approval Moderation were both grayed out for us.  I did a dump of these two groups by running the following two commands:

Distribution Group Mail Flow Message Moderation

Membership Approval Moderation

The Fix

After taking a look at both .txt files, I noticed that the user that was specified under ManagedBy no longer exists.  I changed the ManagedBy to an existing user and all the Moderation feature options lit up and were no longer grayed out.

Keep in mind, you absolutely will need to have a Manager specified in order for the moderation features to light up.  This makes sense with the Membership Approval tab as it only allows you to use Group Managers.  For the Message Moderation, it would seem that Microsoft should not gray everything out by default when you don’t have a Group Manager specified since it does allow you to specify specific people for moderation instead of using the Group Manager.

You can use the following command to modify all your groups in bulk:

Update (1/20/2010) - I ran into another issue today where the moderation options were grayed out even with a valid group manager.  I changed the manager and changed it back and the moderation features were lit up.

Update (2/17/2010) – I forgot to update this with another issue after my 1/20/2010 update.  After updating the Group Manager for a group, the Address Book service in OCS seems to consider this group as an entirely new group.  This means that if people had this group added in OCS and you modified the Group Manager, that group becomes void in Communicator and stops expanding.  I had to end up deleting the GalContacts.db and let a client download a new GalContacts.db.  The client had to delete the group from Communicator and then delete it in order for the group to start working again.

The following data will ultimately need to be backed up:

1. Global Config
2. Pool Config
3. Machine Config
4. SQL Databases
5. Standard Edition File Shares

The first command specifies the /level to be global and pool.  The second command specifies the /level to be machine.  What we will do is create a batch file (.bat) and place both commands in this .bat and have them run against the server every 6pm using scheduled tasks.

Our Servername is SHUD-OCSFE01.  The folder to store the backups is C:\OCSBackup.  We’ll also be running the batch file from the C:\OCSBackup.  Because the folder which contains lcscmde.exe is not a part of the system variables, we’ll have to specify the entire path for lcscmd.exe. Taking this information into consideration, our two commands for our batch file will be:

After executing this .bat file, we can see the two files have been created.

SQL Databases

The following is the list of SQL Databases that an OCS Standard Edition Front End uses:

Because we are utilizing SQL Express, we will have to find some other method other than a backup agent to automate the backups. Much of the SQL Backup information is provided by the SQLDBATips Blog.  The following article I utilized is located here.

Create a file with the extension of sql in our OCSBackup folder.  Also, create a new folder called C:\Reports for script reporting. I created a file C:\OCSBackup\ocssqlbackup.sql with the following text:

exec expressmaint
@database      = ‘ALL_USER’,
@optype        = ‘DB’,
@backupfldr    = ‘c:\ocsbackup’,
@reportfldr    = ‘c:\reports’,
@verify        = 1,
@dbretainunit  = ‘days’,
@dbretainval   = 1,
@rptretainunit = ‘weeks’,
@rptretainval  = 1,
@report        = 1

exec expressmaint
@database      = ‘ALL_USER’,
@optype        = ‘LOG’,
@backupfldr    = ‘c:\ocsbackup’,
@reportfldr    = ‘c:\reports’,
@verify        = 0,
@dbretainunit  = ‘days’,
@dbretainval   = 1,
@rptretainunit = ‘days’,
@rptretainval  = 1,
@report        = 1

All of our OCS Databases are User Databases, not System Databases.  We can see this using SQL Server Management Studio which is not installed by default but can be downloaded from here.

Note: Keep in mind that we’re not using the default SQL Express instance of SQLExpress.  The OCS Front End Standard install will create and utilize an instance of RTC.

We now have our .SQL file created.  We’ll go ahead and create a new .bat file called ocssqlbackup.bat.  This batch file will run the following command:

This won’t work just yet.  You can see in the .SQL file, it’s calling the stored procedure “expressmaint.”  We need to create this stored procedure within SQL.  SQLDBATips has the vbscript code in order to do that here.  You take this code and save it as storemaint.sql.  Then run the following code:

Note: The website that shows these instructions specify the -S.\ as -S.\SQLExpress.  Again, we’re not using the SQLExpress instance, but rather the RTC instance.

You can delete the expressmain.sql file now.  This is a permanent change in our instance and we won’t need to run the expressmain.sql script again.

We should now be able to run our SQL backup batch file as our .sql command that specifies our databases and logs has been created and our batch file to call sqlcmd.exe to execute our .sql file has been created.

We can see our ocssqlbackup.bat file successfully runs and creates backups of our databases.

Scheduled Tasks

We obviously want to keep backing up our databases every night in case something goes wrong.  We’ll create two scheduled tasks.  One that runs ocsbackup.bat for our global, pool, and machine specific information.  And the other that runs our SQL Backups.

I am launching the Task Scheduler from Server Manager (I am using Server 2008 but you can access Task Scheduler on Windows 2003 by going to Control Panel).

Create a Basic Task and give it a name.  We’ll name this OCS Backup.  Click Next to Continue.

Specify how often you want the task to run.  I typically run it Daily. Utilize whatever method works best for your organization. Click Next to Continue.

Choose what time the Daily Task will run.  Again, choose whatever time works best for your organization. Click Next to Continue.

We’ll want to run the script.  Because of this, choose “Start a program.” Click Next to Continue.

Specify the path to our batch file. Click Next to Continue. Review the Settings and then Click Finish.

You can then forcefully run the Scheduled Task to ensure it runs.

Now don’t forget to create the second scheduled task to run the batch file for SQL Backups!

Your OCSBackup folder should look something like this after your scheduled tasks run and your data is backed up.

Backing up your data to a remote Backup Server

Now what good is having all this data backed up onto the OCS File System if OCS crashes?  No good!  We’ll still want to take your backup system and back up all these files including the OCS Standard Edition File Shares.  Now keep in mind that you will want to back up all of these files at some time after your batch files are set to run in Scheduled Tasks.  For example, my Scheduled Tasks are set to run at 8pm.  The batch files do not take long to run.  You can have your backup set to run at 8:30pm or 9:00pm.  Be sure to test and validate this is working as intended and you are getting successful backups.

The Standard Edition File Shares you will want to backup include:

So to sum it up, you will want back up all the above file locations and your OCSBackup folder.  Backing up your Reports folder is optional. But again, keep in mind you will want to run this file level backup after all your Scheduled Tasks are successfully run.