Controlling Configuration Manager Clients Across WAN Links
At work this week, we encountered an issue when a package I created for Adobe Reader 10 went mandatory in Configuration Manager. We service retail stores connected via slow WAN links back to our head offices. When I first joined the company, on a monthly basis when new Windows Updates were released into the wild, our network team would come down upon our team, fire wielding whilst we raped and pilaged the lines to stores.
Configuration Manager gives you the power to create BITS (Background Intelligent Transfer Service) policies to throttle the bandwidth that will be consumed by SCCM client transfers for packages and patches, however the problem with Configuration Manager is that it’s policy is not granular, it is singular which means to apply a policy of 32KB/s which we needed to do to facilitate stores, you would also be limiting the speed of head office clients connected to 100 Megabit or 1 Gigabit high speed LAN connections.
Group Policy also gives you the ability to configure BITS throttling policies and in actual fact, Group Policy gives you more options to control the granularity plus the fact that you can link Group Policies to OUs and not just entire domains, or sites, allows us to control the speeds in a more appropriate way.
In a Group Policy Editor window from Group Policy Management Console (GPMC), navigate to Computer Configuration, Administrative Templates, Network, Background Intelligent Transfer Services (BITS).Â From here, enable the Limit the Maximum Network Bandwidth for BITS Background Transfers setting and configure the speeds and times as you need. You can also configure an out of hours policy which we make use of, limiting the store clients to 32KB/s between 8am and 6pm daily, but allowing them to expand to 256KB/s overnight when the store is closed, not making VoIP calls or trying to transact credit cards.
This worked great and the next time we deployed patches we had no complaints from the stores, but instead, we had a problem at the head office. The problem had not been manifested previously as we had to delay patch deployments before the packages reached everyone, but the issue we experienced now was that due to the length of time a Configuration Manager client was connected to the Distribution Point downloading packages, we were now seeing prolonged connections to the IIS site on the Distribution Point and lots of 64KB/s connections makes lots of bandwidth – We were now actually consuming all of the bandwidth at the head office site, causing inter-site applications between our two head offices to crawl.
We found a solution to this problem in IIS. The solution probably isn’t recommended by Microsoft or any System Center consultants out in the wild, but it works for us and causes no side-effects that we’ve witnessed in the year or so that it has been in place, so it has withstood the test of time.
Using IIS Manager on the Distribution Point server, expand the Default Web Site. From the Actions pane on the right hand side of the interface, select Advanced Settings. From the Advanced Settings dialog, expand the Connection Limits group under the heading Behaviour. By default, IIS accepts a number of connections so large is may as well be infinite. We calculated that based on our free capacity on our link at the head office, taking into account the traffic required for LoB applications, we could handle about 20Mbps of client traffic. We divided the 20Mbps into 64KB/s BITS setting out which gave us a number of 320. Setting the IIS connection limit to 320 and restarting the site in IIS, we saw an instant reduction in Distribution Point server network activity and also the drop we needed on the site link.
As I have mentioned above, this was done over a year ago. In this time, we’ve had not a single complaint from stores, head office users or our network team that there are network contention issues due to SCCM traffic, nor have we seen any apparent SCCM client issues such as clients never properly reporting due to connection retry limits being reached. This isn’t to say that this fix is for everyone: You would need to factor in things like how many total clients do you have in an SCCM site, and if a client was back of the IIS queue, would it be waiting for a connection for so long that it would timeout, or do you need to be able to rapidly deploy high threat updates regardless of the impact to other LoB applications?
Since implanting these changes, we’ve had two Microsoft SCCM RAP assessments and neither have produced red or amber health status problems due to the changes to BITS and IIS, so I think we found ourselves a winner.