Last week a user reported having reset his password, but it hadn’t changed in the connected HR system.
As this is an indication that the Password Change Notification Service (PCNS) wasn’t working, I checked the Event Viewer on the Synchronization Engine server. While I saw several Event IDs indicating that heartbeats were being received from the DCs, there were no Event IDs 6903 for the past several hours. 6903 is the event that indicates a password notification was received from PCNS.
For some reason, password changes were not being sent to the synchronization engine.
In this environment, we have a custom workflow that allows the help desk to reset a user’s password. By turning on logging for that workflow, I could pinpoint the DC that the password change request was being sent to. With the help of a member of the IT Ops group, we checked that Domain Controller and verified that the PCNS service was running.
When we looked in the Event Viewer for that DC, there were no PCNS messages at all—no errors or informational events.
A bit flummoxed at this point, we restarted the PCNS service and saw an error 6032–“The filter configuration for target ‘xxxxx’ is invalid. This will result in ALL notifications being filtered for this target. Verify the target object has correct values for the Inclusion and Exclusion Security Identifiers (SID’s).”
This seemed odd as there had been no changes to this configuration in months. The message did seem to indicate that FIM was unable to find either the inclusion group or the exclusion group.
After a little digging, we found that the Exclusion group was missing. It turns out that during a maintenance process, all groups with no members were deleted and it just so happened the exclusion group was empty (new lesson learned: always put a description in the inclusion and exclusion groups to let administrators know not to delete them—often in a large organization, not all administrators will be aware what the group is being used for).
Since we didn’t need an exclusion group, we tried running pcnscfg to set PCNS to not need an exclusion group with this command:
pcnscfg MODIFYTARGET /N:fimsyncsvc.domain.com /A:fimsyncsvc.domain.com /D:PCNSCLNT/fimsyncsvc.domain.com /FI:“Domain Users” /FE:
Keeping the value to the /FE: parameter empty should remove the exclusion group from the configuration, but in our case we got an error that the name could not be resolved.
Since we knew the exclusion group was missing, we figured it was trying to find the group to commit the change to remove it.
To fix this, we created a new security group for the Exclusion group and ran the PCNSCFG command specifying that group. My assumption is that the MODIFYTARGET command can overwrite the exclusion group value even if it cannot resolve the previous name, but that it cannot clear the exclusion group value in that case.
After specifying the new exclusion group and restarting the service, the password synchronizations started to flow again.
Since this configuration error prevented all password changes from being delivered to the target, there was no queue of password change requests to go to the synchronization server. We therefore only saw them coming in as users changed their passwords again.
Finally, we re-ran the command with the empty /FE: parameter, restarted the PCNS service and password synchronizations flowed again so we were able to delete the Exclusion group.
