Sysadminsblog.com Anything for sysadmins!

2May/130

Dir Sync: Unable to establish a connection to the authentication service

Users reported that they couldn't access their personal archives. The archives are stored on the Office 365 services and should always be accessible. Of course the first clue was located in the event viewer where I found the following errors.

Log Name:      Application
</p><p>Source:        Directory Synchronization
</p><p>Date:          2-5-2013 8:31:38
</p><p>Event ID:      0
</p><p>Task Category: None
</p><p>Level:         Error
</p><p>Keywords:      Classic
</p><p>User:          N/A
</p><p>Computer:      &lt;servername&gt;
</p><p>Description:
</p><p>Unable to establish a connection to the authentication service. Contact Technical Support. GetAuthState() failed with -2147186688 state. HResult:0. Contact Technical Support.  (0x80048862)
</p><p>Event Xml:
</p><p>&lt;Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"&gt;
</p><p>  &lt;System&gt;
</p><p>    &lt;Provider Name="Directory Synchronization" /&gt;
</p><p>    &lt;EventID Qualifiers="0"&gt;0&lt;/EventID&gt;
</p><p>    &lt;Level&gt;2&lt;/Level&gt;
</p><p>    &lt;Task&gt;0&lt;/Task&gt;
</p><p>    &lt;Keywords&gt;0x80000000000000&lt;/Keywords&gt;
</p><p>    &lt;TimeCreated SystemTime="2013-05-02T06:31:38.000000000Z" /&gt;
</p><p>    &lt;EventRecordID&gt;27537&lt;/EventRecordID&gt;
</p><p>    &lt;Channel&gt;Application&lt;/Channel&gt;
</p><p>    &lt;Computer&gt;&lt;servername&gt;&lt;/Computer&gt;
</p><p>    &lt;Security /&gt;
</p><p>  &lt;/System&gt;
</p><p>  &lt;EventData&gt;
</p><p>    &lt;Data&gt;Unable to establish a connection to the authentication service. Contact Technical Support. GetAuthState() failed with -2147186688 state. HResult:0. Contact Technical Support.  (0x80048862)&lt;/Data&gt;
</p><p>  &lt;/EventData&gt;
</p><p>&lt;/Event&gt;

Log Name:      Application
</p><p>Source:        FIMSynchronizationService
</p><p>Date:          2-5-2013 8:31:50
</p><p>Event ID:      6803
</p><p>Task Category: Management Agent Run Profile
</p><p>Level:         Error
</p><p>Keywords:      Classic
</p><p>User:          N/A
</p><p>Computer:      &lt;servername&gt;
</p><p>Description:
</p><p>The management agent "TargetWebService" failed on run profile "Full Confirming Import" because the server encountered errors.
</p><p>Event Xml:
</p><p>&lt;Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"&gt;
</p><p>  &lt;System&gt;
</p><p>    &lt;Provider Name="FIMSynchronizationService" /&gt;
</p><p>    &lt;EventID Qualifiers="49152"&gt;6803&lt;/EventID&gt;
</p><p>    &lt;Level&gt;2&lt;/Level&gt;
</p><p>    &lt;Task&gt;1&lt;/Task&gt;
</p><p>    &lt;Keywords&gt;0x80000000000000&lt;/Keywords&gt;
</p><p>    &lt;TimeCreated SystemTime="2013-05-02T06:31:50.000000000Z" /&gt;
</p><p>    &lt;EventRecordID&gt;27539&lt;/EventRecordID&gt;
</p><p>    &lt;Channel&gt;Application&lt;/Channel&gt;
</p><p>    &lt;Computer&gt;&lt;servername&gt;&lt;/Computer&gt;
</p><p>    &lt;Security /&gt;
</p><p>  &lt;/System&gt;
</p><p>  &lt;EventData&gt;
</p><p>    &lt;Data&gt;TargetWebService&lt;/Data&gt;
</p><p>    &lt;Data&gt;Full Confirming Import&lt;/Data&gt;
</p><p>  &lt;/EventData&gt;
</p><p>&lt;/Event&gt;

Log Name:      Application
</p><p>Source:        FIMSynchronizationService
</p><p>Date:          2-5-2013 8:31:50
</p><p>Event ID:      6110
</p><p>Task Category: Management Agent Run Profile
</p><p>Level:         Warning
</p><p>Keywords:      Classic
</p><p>User:          N/A
</p><p>Computer:      &lt;servername&gt;
</p><p>Description:
</p><p>The management agent "TargetWebService" step execution completed on run profile "Full Confirming Import" but the watermark was not saved.
</p><p> 
 </p><p> Additional Information
</p><p> Discovery Errors       : "0"
</p><p> Synchronization Errors : "0"
</p><p> Metaverse Retry Errors : "0"
</p><p> Export Errors          : "0"
</p><p> Warnings               : "0"
</p><p> 
 </p><p> User Action
</p><p> View the management agent run history for details.
</p><p>Event Xml:
</p><p>&lt;Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"&gt;
</p><p>  &lt;System&gt;
</p><p>    &lt;Provider Name="FIMSynchronizationService" /&gt;
</p><p>    &lt;EventID Qualifiers="32768"&gt;6110&lt;/EventID&gt;
</p><p>    &lt;Level&gt;3&lt;/Level&gt;
</p><p>    &lt;Task&gt;1&lt;/Task&gt;
</p><p>    &lt;Keywords&gt;0x80000000000000&lt;/Keywords&gt;
</p><p>    &lt;TimeCreated SystemTime="2013-05-02T06:31:50.000000000Z" /&gt;
</p><p>    &lt;EventRecordID&gt;27540&lt;/EventRecordID&gt;
</p><p>    &lt;Channel&gt;Application&lt;/Channel&gt;
</p><p>    &lt;Computer&gt;&lt;servername&gt;&lt;/Computer&gt;
</p><p>    &lt;Security /&gt;
</p><p>  &lt;/System&gt;
</p><p>  &lt;EventData&gt;
</p><p>    &lt;Data&gt;TargetWebService&lt;/Data&gt;
</p><p>    &lt;Data&gt;Full Confirming Import&lt;/Data&gt;
</p><p>    &lt;Data&gt;0&lt;/Data&gt;
</p><p>    &lt;Data&gt;0&lt;/Data&gt;
</p><p>    &lt;Data&gt;0&lt;/Data&gt;
</p><p>    &lt;Data&gt;0&lt;/Data&gt;
</p><p>    &lt;Data&gt;0&lt;/Data&gt;
</p><p>  &lt;/EventData&gt;
</p><p>&lt;/Event&gt;

When I started the Synchronization Service Manager at

C:\Program Files\Microsoft Online Directory Sync\SYNCBUS\Synchronization Service\UIShell\miisclient.exe

, it showed the error stopped-server-down.

Seeing that it couldn't connect to some server I had to find out what server it was trying to connect to. In the Synchronization Service Manager I checked the Management Agents where in the properties of the TargetWebService I found the server that it was trying to connect to; https://adminwebservice.microsoftonline.com/ProvisioningService.svc.

After clearing the cache of the DNS services and flushing the DNS locally on the server I forced another full import which ran without problems.

Hope this helps you with solving your problem!

12Dec/120

Mailbox size overview for Exchange 2010

Once every now and then you'll want an overview of what the mailbox sizes are for your organization. You can then see who is using the most space and perhaps offer them some help archiving their mailbox. However Exchange 2010 doesn't really offer this in a organized way. This is exactly the reason why I created this one-liner. This will output to a GridView or to a CSV which you can then use to organize the data as you wish.

Get-MailboxDatabase | % { Get-MailboxStatistics -Database $_.Name | Select DisplayName, @{Name="Size (MB)";Expression={($_.TotalItemSize).Substring($a.IndexOf("(")).Replace("(","").Replace(" bytes)","").Replace(",","") / 1MB}} }

You could output it to GridView by piping it to

Out-GridView

Get-MailboxDatabase | % { Get-MailboxStatistics -Database $_.Name | Select DisplayName, @{Name="Size (MB)";Expression={($_.TotalItemSize).Substring($a.IndexOf("(")).Replace("(","").Replace(" bytes)","").Replace(",","") / 1MB}} } | Out-GridView

Or export it to a CSV by piping it to

Export-CSV

:

 Get-MailboxDatabase | % { Get-MailboxStatistics -Database $_.Name | Select DisplayName, @{Name="Size (MB)";Expression={($_.TotalItemSize).Substring($a.IndexOf("(")).Replace("(","").Replace(" bytes)","").Replace(",","") / 1MB}} } | Export-Csv C:\MailboxSizes.csv

As a result you can open the CSV in Excel, add some Conditional Formatting and Column Formating and voila! Yes, these are the actual mailbox sizes!

7Dec/123

PST Capture errors

I'm currently in the process of configuring PST Capture in order to migrate all the user's PST files to the recently configured cloud based Online Archive. As this wasn't working properly, I contacted Microsoft and opened a support ticket. I'm getting the error "Import error: Error opening mailbox <mail address>" when I try to import the PST file into my Online Archive. During the troubleshooting of this error I came across a nice list of requirements for the PST Capture tool to function properly. This might help you to configure everything correctly.

  1. Please make sure that you have the 64 Bit version of Outlook installed and that it is installed on the machine on which PST Capture Tool is running.
  2. Please check if the user has the Mailbox Import Export permissions. If it does not have them, then please assign them using the following command:
    </em>New-ManagementRoleAssignment -Role "Mailbox Import Export" -User "PSTImportUser"
  3. Please check that the Microsoft Exchange PST Capture Tool's Service Account has an Exchange 2010 Mailbox.
  4. Please check that the PST Capture Tool's Service Account is a member of Local Administrators Group.
  5. Please make sure that the PST Capture Tool's Service Account is a member of Organization Management and also the Public Folder Management.
  6. Check that the Service Account is mailbox-enabled and has an Outlook profile configured for it.
  7. Please make sure that the correct Service Account is chosen for the PST Capture Tool. To do this, please go to Properties of 'Microsoft Exchange PST Capture Service' > Log on tab  and check the Service Account.

 

Please comment below if you have questions about the PST Capture tool, or the above settings.

21Jun/125

Room List Distribution Group

I just came across another Exchange 2010 surprise that I would like to share with you. When you are creating a new meeting and need a room to have the meeting in, you would normally add all the rooms and see which one is available. The guys behind Exchange 2010 came up with a solution for this. They introduced the Room List.

The room list is essentially a distribution group but with an added parameter. Unfortunately you can't create a Room List in the Exchange Management Console, so you'll have to use the Exchange Management Shell (PowerShell) for this. The command is quite straight forward.

New-DistributionGroup -Name "Meeting Rooms" –Members ConfRoom1,ConfRoom2,ConfRoom3,ConfRoom4 -RoomList

As you can see, the -RoomList parameter makes all the difference!

People no longer need to sift through the meeting rooms manually. They can now effectively use the Room Finder to find a suitable room.

 

16May/120

Unknown Start Trace Error (183) when starting Exmon

I recently used the Exchange User Monitor tool to check the connections because the exchange server was running at 100% CPU constantly. When Exmon.exe crashed I weren't able to restart it again as it presented an error. A quick Google query pointed me in the right direction.

An Exchange trace is initiated when the Exmon.exe tool is started. The trace is not closed properly when the tool crashes and it has to be closed manually.

  1. Make sure that Exmon.exe is not running (use CTRL+SHIFT+ESC)
  2. From a CMD run:
    logman query -ets

    This will show you all tracers that are currently running. Notice the Exchange Event Trace that's used by Exmon

  3. Stop the Exchange Event Trace:
    Logman stop "Exchange Event Trace" –ets

If that didn't resolve the issue, check if there's still some space left on the disk or if another user is already running the Exmon tool.

18Aug/110

Mailbox Import/Export Exchange cmdlets unavailable

The <verb>-MailboxExportRequest and <verb>-MailboxImportRequest cmdlets have been introduced in Exchange 2010 SP1 to simplify the export of messages directly into a PST file from PowerShell. The previous requirement to have Outlook and some other software installed on the exporting computer have been dropped and the whole export process has been moved to the Mailbox Replication Service (MRS). A small disadvantage is that the cmdlets are not available by default as you'll first need to define the proper permission to your role group.

I can write in detail how to add these permissions, but Microsoft has done this quite well already in this technet article: Add the Mailbox Import Export Role to a Role Group

This command will get you started:

New-ManagementRoleAssignment -Name "Import Export_Enterprise Support" -SecurityGroup "Enterprise Support" -Role "Mailbox Import Export"

18Aug/110

Unable to open PST file with mailbox import/export requests

I was trying to an export of a mailbox using the New-MailboxExportRequest CMDlet in Exchange 2010 SP1. However instead of creating a PST it gave me an error.

Unable to open PST file '\\Server\Exports\Test.pst'. Error details: Access to the path '\\ExServer1\Imports\Test1.pst' is denied.;

Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: Access to the path '\\Server\Exports\Test.pst' is denied.

I was soon pointed into the right direction by a blog post by Tony Redmond. He indicated that because the Mailbox Replication Server (MRS) is running as LocalSystem it can't access a network share. By adding the Exchange Trusted Subsystem group to the share permissions you will give the LocalSystem account and therefore MRS access to the share.

19Jul/110

Slow mail delivery to Outlook client from Exchange 2010

I just had a lot of problems with email being delivered to my Outlook 2010 really slow. The following findings are the result of my troubleshooting:

  • Messages are delayed ranging from a couple of minutes up to an hour when using Outlook 2010 (using RPC or Outlook Anywhere)
  • Messages are not delayed in OWA
  • PowerShell shows the messages being delivered into the proper folder (Get-MailboxFolderStatistics)
  • Message tracking shows that the internal transport is not delayed
  • Reconnecting your Outlook will force the download of the delayed messages (CTRL+Right click the taskbar icon, selecting Connection status)
  • Messages are not delayed using ActiveSync

After hitting Google with these findings it soon pointed me to the problem and also the solution.

RPC traces showed that the server couldn't contact the clients somehow.

The solution

Install Exchange 2010 SP1 Update Rollup 3 (v3)

Description of Update Rollup 3 for Exchange Server 2010 Service Pack 1

Download Update Rollup 3 for Exchange Server 2010 Service Pack 1

The installation of the Update Rollup will require a reboot of the Exchange server, but it will solve this particular issue along with other issues.

The article that pointed me in the right direction and also the source of some of the troubleshooting steps can be found right here.

25Nov/100

Database Maintenance in Exchange 2010 – Online Defragmentation, Monitoring, Whitespace

Online Defragmentation/Monitoring

With the coming of Exchange 2010 the database maintenance has been simplified a lot. No more worries about online maintenance windows overlapping with the database backups. The reason for this is that the online defragmentation has been moved out of the mailbox database maintenance process. The online defragmentation is now running in the background all the time.

This option is enabled by default and no further settings are required. You can monitor the database defragmentation with the performance monitor by adding a couple of performance counters.

  • MSExchange Database ==> Instances \ Defragmentation tasks        Shows the background database defragmentation tasks currently executing.
  • MSExchange Database ==> Defragmentation Tasks completed/Sec    Shows the number of background database defragmentation tasks completing execution per second.
  • MSExchange Database ==> Defragmentation Tasks Discarded        Shows the background database defragmentation tasks that couldn't be registered.
  • MSExchange Database ==> Defragmentation Tasks Pending        Shows the background database defragmentation tasks currently pending.
  • MSExchange Database ==> Instances \ Defragmentation        Tasks Scheduled/Sec Shows the background database defragmentation tasks scheduled for execution per second.

The performance counters are solely to check performance and are not required to be part of the daily maintenance of the database.

It's also possible to enable extended Extensible Storage Engine (ESE) performance counters, but to enable these counters you'll have to change a setting in the registry.

Whitespace

If you want to track the whitespace in your Exchange 2010 database you can use the following EMC command:

Get-MailboxDatabase -Status | Select-Object Name,AvailableNewMailboxSpace

The returned amount is however only the space available in the ESE B+-tree structure.

Older versions of Exchange generate an event with ID 1221 to show the whitespace. Exchange 2010 generates a similar event only for the mail queue database with event ID 7007.

During my search for information about whitespace I came across an interesting script by Mike Pfeiffer that generates quite some useful data about the mailbox database.

 Function Get-DatabaseStatistics {
</p><p>    $Databases = Get-MailboxDatabase -Status
</p><p>    ForEach($Database in $Databases) {
</p><p>        $DBSize = $Database.DatabaseSize
</p><p>        $MBCount = (Get-Mailbox -Database $Database.Name).Count
</p><p>        $MBAvg = $DBSize.ToBytes() / $MBCount            
</p><p>        New-Object PSObject -Property @{
</p><p>            Server = $Database.Server.Name
</p><p>            DatabaseName = $Database.Name
</p><p>            LastFullBackup = $Database.LastFullBackup
</p><p>            MailboxCount = $MBCount
</p><p>            "DatabaseSize (GB)" = "{0:n2}" -f ($DBSize.ToBytes() / 1GB)
</p><p>            "AverageMailboxSize (MB)" = "{0:n2}" -f ($MBAvg / 1MB)
</p><p>            "WhiteSpace (MB)" = "{0:n2}" -f ($Database.AvailableNewMailboxSpace.ToBytes() / 1MB)
</p><p>        }
</p><p>    }
</p><p>}

After running this script in the EMC you can execute the script by running Get-DatabaseStatistics. The output should be similar to the image below.

24Sep/107

Event 1007 – MSExchange Mailbox Replication

The Mailbox Replication service was unable to determine the set of active mailbox databases on a mailbox server.<br/>Mailbox server: Exchangeserver.spil.local<br/>Error: MapiExceptionNetworkError: Unable to make admin interface connection to server. (hr=0x80040115, ec=-2147221227)<br/>Diagnostic context:<br/>    ......<br/>    Lid: 15000   dwParam: 0x6BA      Msg: EEInfo: prm[1]: Pointer val: 0x0000000000000000<br/>    Lid: 15000   dwParam: 0x6BA      Msg: EEInfo: prm[2]: Pointer val: 0x985CA8C000000000<br/>    Lid: 16280   dwParam: 0x6BA      Msg: EEInfo: ComputerName: n/a<br/>    Lid: 8600    dwParam: 0x6BA      Msg: EEInfo: ProcessID: 2256<br/>    Lid: 12696   dwParam: 0x6BA      Msg: EEInfo: Generation Time: 2010-09-24 11:32:34:750<br/>    Lid: 10648   dwParam: 0x6BA      Msg: EEInfo: Generating component: 18<br/>    Lid: 14744   dwParam: 0x6BA      Msg: EEInfo: Status: 10060<br/>    Lid: 9624    dwParam: 0x6BA      Msg: EEInfo: Detection location: 318<br/>    Lid: 13720   dwParam: 0x6BA      Msg: EEInfo: Flags: 0<br/>    Lid: 11672   dwParam: 0x6BA      Msg: EEInfo: NumberOfParameters: 0<br/>    Lid: 24060   StoreEc: 0x80040115<br/>    Lid: 23746  <br/>    Lid: 31938   StoreEc: 0x80040115<br/>    Lid: 19650  <br/>    Lid: 27842   StoreEc: 0x80040115<br/>    Lid: 20866  <br/>    Lid: 29058   StoreEc: 0x80040115

This seems to be related to the configuration of the NIC(s) in your Exchange server. I've seen a couple of causes for this problem. Confirm all of them are correct and the issue should be resolved.

  • Using DHCP for one or more NICs. Setting this NIC to static might solve the problem.
  • Setting multiple IP addresses on one NIC. Editing the DNS to only refer to 1 IP instead of both might solve the problem.
  • Having a disabled NIC set to DHCP. Editing the disabled NIC to use static IP might solve the problem.
  • Information Store service is not running. Start the Information Store service.
  • IPv6 is enabled and you're not using it. Disable IPv6.

There are a couple of other possible solutions, but those have not been confirmed to work yet. If you found another possible solution, please comment below!

Interesting read: