Applying software GPO takes 20 minutes and times out on RODC site

There exists an annoying bug in Windows 7 with GPOs and RODCs that makes your users waiting 20 minutes until the computer starts installing software via Active Directory.

Symtoms:

  • The infrastructure has a central site with 2 DCs 2008 R2 and several branch offices with Read Only Domain Controllers located on every office. The problem is that when some changes are done on the Group Policies in the central site (modify old GPOs, create new ones, most often software installations) after the restart (or gpupdate /force) when the new settings are applied - it take about 20 minutes for the client to boot.
  • Windows boots and show a throbber with "Applying software installation policy..." for about 20 minutes (10 minutes per machine + 10 minutes for user GPO timeout). This happens only after you have changed a GPO and only once after a change. It doesn't matter if this is a software policy or any random setting. The setting get's applied and all other future reboots are fine, until you change anything again inside a GPO.
  • gpupdate /forcetakes 20 minutes in a sum, until it times out applying per machine GPOs and per user GPOs.
  • Debugging GPO shows no progress within this 20 minutes. You will see the GPO process starts and than 10 minutes nothing logged in the logfiles.
  • The affected computers are all in a site with a Read Only Domain Controller (RODC). Computers in sites with a writeable Domain Controller are not affected.
  • After these 20 minutes the software installations run normally and the login screen appears afterwards as expected.
  • Users need to wait 20 minutes + installation time until they can log on to their computer.
  • Wireshark network traces show you high random LDAP traffic.
  • Eventlog messages with EventID 6006 The winlogon notification subscriber took X second(s) to handle the notification event (CreateSession). are logged.

Details:

This issue has a very long history with very confusing and illogical analysis steps from MS. They told us the RODCs NIC drivers with failover configured, virus scanners and other unreleated stuff are known to make troubles. This was all plain wrong and nothing changed with any of the hotfixes we have been forced to try out:

You can skip all this stuff and focus on the real source. While moving the client to a test OU by blocking all GPOs the problems have gone away. We added one policy after the other back and after adding some GPO policies the issue started again. We have been so luckily to find a logic as every added software policy has added additional 2 minutes to the "Applying software installation policy..." throbber, but the first policy has not added this 2 minutes delay.

Repro case:

  1. Move a client to it's own test OU.
  2. Block policy inheritance and only add the GPOs you really cannot run without, but no software policies.
  3. Add first software policy to client OU, no boot delay
  4. Add second software policy to client OU; The winlogon notification subscriber took 123 second(s) to handle the notification event (CreateSession).
  5. Add third software policy to client OU; The winlogon notification subscriber took 224 second(s) to handle the notification event (CreateSession).
  6. Add fourth software policy to client OU; The winlogon notification subscriber took 360 second(s) to handle the notification event (CreateSession).

You cannot expect that MS will document this very soon or at all and this is why this article get's published.

Debugging documentation:

Resolution:

History:

Rating
Average: 7.9 (28 votes)