Controlling Configuration Manager Clients Across WAN Links

At work this week, we encountered an issue when a package I created for Adobe Reader 10 went mandatory in Configuration Manager. We service retail stores connected via slow WAN links back to our head offices. When I first joined the company, on a monthly basis when new Windows Updates were released into the wild, our network team would come down upon our team, fire wielding whilst we raped and pilaged the lines to stores.

Configuration Manager gives you the power to create BITS (Background Intelligent Transfer Service) policies to throttle the bandwidth that will be consumed by SCCM client transfers for packages and patches, however the problem with Configuration Manager is that it’s policy is not granular, it is singular which means to apply a policy of 32KB/s which we needed to do to facilitate stores, you would also be limiting the speed of head office clients connected to 100 Megabit or 1 Gigabit high speed LAN connections.

Group Policy also gives you the ability to configure BITS throttling policies and in actual fact, Group Policy gives you more options to control the granularity plus the fact that you can link Group Policies to OUs and not just entire domains, or sites, allows us to control the speeds in a more appropriate way.

In a Group Policy Editor window from Group Policy Management Console (GPMC), navigate to Computer Configuration, Administrative Templates, Network, Background Intelligent Transfer Services (BITS). From here, enable the Limit the Maximum Network Bandwidth for BITS Background Transfers setting and configure the speeds and times as you need. You can also configure an out of hours policy which we make use of, limiting the store clients to 32KB/s between 8am and 6pm daily, but allowing them to expand to 256KB/s overnight when the store is closed, not making VoIP calls or trying to transact credit cards.

This worked great and the next time we deployed patches we had no complaints from the stores, but instead, we had a problem at the head office. The problem had not been manifested previously as we had to delay patch deployments before the packages reached everyone, but the issue we experienced now was that due to the length of time a Configuration Manager client was connected to the Distribution Point downloading packages, we were now seeing prolonged connections to the IIS site on the Distribution Point and lots of 64KB/s connections makes lots of bandwidth – We were now actually consuming all of the bandwidth at the head office site, causing inter-site applications between our two head offices to crawl.

We found a solution to this problem in IIS. The solution probably isn’t recommended by Microsoft or any System Center consultants out in the wild, but it works for us and causes no side-effects that we’ve witnessed in the year or so that it has been in place, so it has withstood the test of time.

Using IIS Manager on the Distribution Point server, expand the Default Web Site. From the Actions pane on the right hand side of the interface, select Advanced Settings. From the Advanced Settings dialog, expand the Connection Limits group under the heading Behaviour. By default, IIS accepts a number of connections so large is may as well be infinite. We calculated that based on our free capacity on our link at the head office, taking into account the traffic required for LoB applications, we could handle about 20Mbps of client traffic. We divided the 20Mbps into 64KB/s BITS setting out which gave us a number of 320. Setting the IIS connection limit to 320 and restarting the site in IIS, we saw an instant reduction in Distribution Point server network activity and also the drop we needed on the site link.

As I have mentioned above, this was done over a year ago. In this time, we’ve had not a single complaint from stores, head office users or our network team that there are network contention issues due to SCCM traffic, nor have we seen any apparent SCCM client issues such as clients never properly reporting due to connection retry limits being reached. This isn’t to say that this fix is for everyone: You would need to factor in things like how many total clients do you have in an SCCM site, and if a client was back of the IIS queue, would it be waiting for a connection for so long that it would timeout, or do you need to be able to rapidly deploy high threat updates regardless of the impact to other LoB applications?

Since implanting these changes, we’ve had two Microsoft SCCM RAP assessments and neither have produced red or amber health status problems due to the changes to BITS and IIS, so I think we found ourselves a winner.

Supported SQL Server Versions for System Center 2012

Following on from my post SQL Disk Space Error Installing SCAC 2012 I contacted Microsoft Premier Support to confirm the supported service pack levels for SQL Server 2008 because the release notes were very wooly as to whether Service Pack 3 was supported as well as the stated Service Pack 2.

Here is the response I recieved from them:

When I sent my request to the product group, the Group Program manager for SC 2012, replied back that those products listed are the only ones that were tested and thus fully supported.

It is entirely possible that you could run with SQL 2008 SP3 with no issues, but if a problem is encountered related to the databases we may request you relocate the database to one of the listed SQL platforms.

Until SQL Server 2008 with Service Pack 3 is recognised as tested and supported, I’d recommend that you just go with Service Pack 2 and install the latest Cumulative Update, which is currently Cumulative Update 9.

SQL Disk Space Error Installing SCAC 2012

When installing SCAC (System Center App Controller) 2012 RTM yesterday, I encountered an error at the SQL database definition step “The Specified Database has Insufficent Disk Space”.

Let it be a lesson to us all to properly review the release notes prior to installing products. http://technet.microsoft.com/en-us/library/gg696060.aspx

SCAC (System Center App Controller) supports the following SQL database servers:

SQL Server 2008 Standard and Enterprise with Service Pack 2
SQL Server 2008 R2 Standard, Enterprise and Datacenter

Suprisingly though, the release note make no mention as to whether service pack levels above those stated are supported (such as SQL Server 2008 with Service Pack 3 or SQL Server 2008 R2 with Service Pack 1). One would assume that they will support any upward versions and that the versions stated as the minimum supported, but you never know?

Typo in SCAC 2012 AppControllerSetupWizard Log

When installing SCAC (System Center App Controller) 2012, you may need to review the installation logs. These are placed in the %UserProfile%\AppData\Local\Temp\AppController\Logs path.

Reviewing the log today, I found a typo in the file. It’s not anything that’s going to effect the installation, but worth noting if you are trying to search the log files for particular keywords.

10:14:02:RemoveConceroRegistry: Removing registry key on rollbak.

This should obviously read rollback not rollbak.

Import Custom Phyiscal Resources in VMM 2012

This evening, I have been working to create some Physical Resource Packages in System Center VMM 2012 for a project at work. VMM by default supports the following file types and extensions for resources:

  • Answer Files (.inf .ini .xml)
  • ISO Images (.iso)
  • Script Files (.ps1 .sql)
  • Virtual Floppy Disks (.flp .vld)
  • Virtual Hard Disk (.vhd .vmdk)

The resource package I am trying to create will be used as part of an Application Profile to allow VMM to install a server-side application as part of a Service Template Deployment, however as the installation file consists of a .msi Windows Installer Package, VMM wouldn’t import it.

Looking at the pre-defined resource packages, I noticed that the Server App-V and the Web Deploy resource packages contained .msi files and they were imported okay, so what was the difference?

The difference is in the handling. VMM by default asks for a folder path where your custom resources reside after which it will import them into a library. By default, once you provide a path, VMM will import all supported objects from path. To add unsupported objects such as .exe and .msi files for Application Profiles, you must append the folder name for your source files with .cr.

This addition causes VMM to import the entire folder regardless of content. In my example, my folder name went from Microsoft_iSCSI_Software_Target to Microsoft_iSCSI_Software_Target.cr and VMM now has the package imported.

Package Fails to Distribute in SCCM When an autorun.inf File is Present

At work this week, I was working with an Intel HD Graphics driver package which in terms of SCCM, you would call a bad driver. We call it a bad driver because it is a driver which doesn’t not install correctly using the Apply Device Drivers OSD step but instead requires a full application to be executed.

After creating the package in SCCM, I proceeded to distribute the package to our distribution points on the network so that the operating system deployment process would be able to access the files required to deploy the application.

After waiting a short while for the package to distribute, I checked the Package Status view in the ConfigMgr Console, and I saw that the status was Install Retrying. After looking at the status log for the distribution point, I saw that it had already gone into a retrying state several times. If received the following error:

SMS Distribution Manager failed to copy package "SITE0011C" from "\SERVERPATHIntelHD Graphics Display Driverx64" to "MSWNET:["SMS_SITE=SITE"]\SERVERSMSPKGD$SITE0011C".

Possible cause: SMS Distribution Manager does not have sufficient rights to read from the package source directory or to write to the destination directory on the distribution point.

Solution: In the SMS Administrator console, verify that the site system connection accounts have sufficient privileges to the source and destination directories.

Possible cause: The distribution point might not be accessible to SMS Distribution Manager running on the site server.

Solution: If users are currently accessing the package files on the distribution point, disconnect the users first. If the package distribution point is located on a Windows NT computer, you can force users to disconnect by clicking on the "Disconnect users from distribution points" box in the Data Access tab of the Package Properties dialog box.

Possible cause: The distribution point does not have enough free disk space to store the package.

Solution: Verify that there is enough free disk space.

Possible cause: The package source directory contains files with long file names and the total length of the path exceeds the maximum length supported by the operating system.

Solution: Reduce the number of folders defined for the package, shorten the filename, or consider bundling the files using a compression utility.

I logged into the effected distribution point and verified that the file shares used by SCCM where still active and that there was sufficient disk space on the server which there was.

I have encountered issues with package distribution before with a Windows 7 64-bit image was refusing to distribute, but I couldn’t find any cause, and in that instance re-creating the package resolved the issue, so my first port of call was this. On the sources directory, I made a new folder and copied the source files from my workstation fresh to the server in case there had been a problem with the previous file transfer.

On this occasion, whilst copying the files, I got an error whilst trying to copy the files, and it specifically generated the error no the autorun.inf file which was included in the download from the Intel site. I thought this was wierd, but knowing how invasive our McAfee enterprise policies can be at times, I wondered if the autorun.inf file was causing an issue. I deleted the autorun.inf file from the original package sources directory on the server and watched while SCCM happily distributed the package to the distribution points.

After a quick bit of investigation, I soon discovered a setting in McAfee VirusScan called Prevent Remote Creation of autorun.inf Files which was enabled. Because SCCM uses SMB to transfer the packages from the source directory to the distribution points, this triggered the McAfee rule and blocked the entire package from being created.

As a rule on thumb, there is no reason to have autorun.inf files inside your SCCM packages and their source directories, so in this instance I simply omitted the file, however if you needed to keep the file, then you could simply disable this protection rule for your SCCM Site and Distribution Point servers and the server which holds your package source files (perhaps a File Server). Although I have mentioed McAfee as the culprit in this scenario, I’m pretty sure that other anti-virus applications will feature a similar rule which could cause you other headaches.

Configuring IIS Redirects for HTTPS with the SCOM 2007 R2 Web Console

Whilst working with the SCOM 2007 R2 Console today, I saw that on our SCOM RMS server, the Default Web Site in IIS was running still and occupying Port 80 for no good reason, while the SCOM Console was relegated to Port 51908 which isn’t very user friendly. Additionally, the site was in the clear with no SSL, so I wanted to make the site secure.

Step 1 was to disable the Default Web Site and stop it from starting automatically. Once this is done, remove the Binding for Port 80 from the site to make that port available for use. Once you have done this, you can follow the steps per my previous post Redirecting Non-HTTPS Traffic to HTTPS for SharePoint 2007. Although the post in entitled for SharePoint 2007, it applies to any server running Windows Server 2008 or 2008 R2 with IIS 7 or 7.5, just you need to rename the websites that you create accordingly.

Once complete, users will be able to type the server name which hosts your SCOM Web Console, without needing to append the default port number, and they will be automatically redirected to Port 443 for the HTTPS version of the site, instead of an IIS error stating that they need to use the HTTPS version.

System Center Operations Manager 2007 R2 Web Console Authentication

Whilst working on something un-related today, i discovered a problem with our SCOM 2007 R2 Web Console at work – When I tried to connect to the site, I was prompted for my credentials and I provided my domain logon, but it kept coming back at me until eventually, I got a HTTP 30 Unauthorised error.

A lot of blogs and forum topics online including some at Microsoft (Example: http://blogs.technet.com/b/kevinholman/archive/2008/09/24/installing-the-web-console-on-a-2008-management-server-using-windows-authentication.aspx) will recommend that you configure Kerberos Delegation for the computer account which hosts the Web Console, using the credentials of the SCOM SDK Service Account.

This is my eyes was a bit of a dirty hack, and the cleanest and most obvious solution had to lie in IIS and its Authentication schemas.

image

Sure as could be, the OPWebConsoleApp Application Pool in IIS was configured with ApplicationPoolIdentity which in English means it’s not no permissions on the network, or has no access to the domain to verify domain credentials.

My solution to the problem is as follows.

Firstly, create a new Application Pool in IIS. Call it what you like, but this will be hosting your Operations Manager Web Console, so best to name it accordingly. I named mine SCOM 2007 R2 Web Console. I also elected not to have the Application Pool start immediately, as we need to configure the credentials on the Pool first.

image

Right-click on the new Pool, and select Advanced Settings. Under the Process Model group, there is an item called Identity – Click the … button on the right of the line to open the next dialog. Change the Identity to Custom Account and specify the username and password for a domain service account which can host the Pool, then click OK button you get back to the Application Pools list in IIS.

Now you can start the Pool by right-clicking and select Start. If the Pool fails to start, you need to verify that your credentials specified for the Pool were correct, and that you don’t have a Group Policy preventing that account from running as a service or such.

Now, right-click on the existing OPWebConsoleApp Applications Pool and select View Applications.

image

Right-click on each of the applications, and select the Change Application Pool option. You will be given a list of available Pools, and select the one which you just created.

Once complete, you need to restart Web Sites, however the easiest thing to do, is from an elevated command prompt type iisreset which will reset all of the Pools and Web Sites.

Assuming you have Windows Authentication enabled on the Operating Manager 2007 WebConsole Web Site (which you should by default) then you should now be able to successfully access the site using Single-Sign On (SSO) with no requirement to enter credentials.

For bonus points, you can be a friendly, security conscious administrator an set the site to Require SSL Encryption and create a new IIS Web Site to redirect Non-SSL users to the SSL site.