Earlier on this week, I finally got around to spending a bit of time towards building my home lab. I know it’s late because I started this project back in February but you know how it is.
On the servers, I am installing Windows Server 2012 R2 with Update which for the uninitiated is KB2919355 for Windows Server 2012 R2 and Windows 8.1. This is essentially a service pack sized update for Windows and includes a whole host of updates. I am using the installation media with the update integrated to same me some time with the updates but also because it’s cleaner to start with the update pre-installed.
The Inaccessible Boot Device Problem
After installing Windows Server 2012 R2, the machine starts to boot and at the point where I would expect to see a message along the lines of Configuring Devices, the machine hits a Blue Screen of Death with the message Stop 0x7B INACCESSIBLE_BOOT_DEVICE and restarts. This happens a few times before it hangs on a black screen warning that the computer has failed to start after multiple attempts. I assumed it was a BIOS problem so I went hunting in the BIOS in case I had enabled a setting not supported by my CPU or maybe I’d set the wrong ACHI or IDE mode options but everything looked good. I decided to try the Optimized Defaults and Failsafe Defaults options in the BIOS, both of which required an OS re-install due to the AHCI changes but neither worked.
After this I was worried there was either something wrong with my hardware or a compatibility issue with the hardware make-up and I was going to be snookered however after a while of searching online, I found the solution.
KB2919355 included a new version of the storage controller driver Storport. It transpires that this new version of Storport in KB2919355 had an issue with certain SCSI and SAS controllers whereby if the controller device driver was initialized in a memory space beyond 4GB then it would cause the phyiscal boot devices to become inaccessible. This problem hit people who installed the KB2919355 update to previously built servers at the time of release as well as people like me, building new servers with the update slipstreamed. My assumption is that it’s caused by the SCSI or SAS controller not being able to address 64-bit memory addresses hence the 4GB limitation.
The problem hits mainly LSI based SCSI and SAS controllers based on the 2000 series chipset, including but by no means limited to the LSI SAS 2004, LSI SAS 2008, LSI MegaRAID 9211, Supermicro SMC 2008, Dell PERC H200 and IBM X240 controllers. In my case, my Supermicro X8DTH-6F motherboards have the Supermicro SMC 2008 8 Port SAS controller onboard which is a Supermicro branded edition of the LSI SAS 2008 IR controller.
The workaround at the time was to disable various BIOS features such as Intel-VT, Hyperthreading and more to reduce the number of system base drivers that needed to load, allowing the driver to fit under the 4GB memory space but eventually the issue was confirmed and a hotfix released however installing the hofix is quite problematic when the system refuses to boot. Luckily, we can use the Windows installation media to fix the issue.
Microsoft later released guidance on the workaround to use BCDEdit from the Windows Recovery Environment (WinRE) to change the maximum memory.
Resolving the Issue with KB2966870
Workarounds aside, we want to fix the issue not gloss over or around it. First off, download the hotfix KB2966870 which is a hotfix by request so you need to enter your email address and get the link emailed to you. You can get the update from https://support.microsoft.com/kb/2966870. Once you have the update, you need to make it available to your server.
If your Windows Server 2012 R2 installation media is a USB bootable stick or drive then copy the file here. If your installation medium is CD or DVD then burn the file to a disc.
Boot the server using the Windows Server 2012 R2 media but don’t press the Install button. From the welcome screen, press Ctrl + F10 which will open a Command Prompt in Administrator mode. Because of the Windows installation files being decompressed to a RAM disk, your hard disk will have likely been mounted on D: instead of C: but verify this first by doing a dir to check the normal file structure like Program Files, Users and Windows. Also, locate the drive letter of your installation media which will be the drive with your .msu update file on it.
Once you have found your hard disk drive letter and your boot media letter, we will use the following DISM command to install the update using Offline Servicing:
Dism /Image:[Hard Disk]:\ /Add-Package /PackagePath:[Install Media]:\Windows8.1-KB2966870-x64.msu
Once the command completes, exit the Command Prompt and exit the Windows Installation interface to restart the computer. In my case, I had to restart the computer twice for the update to appear to actually apply and take effect but once the update had been taken on-board, the machine boots without issues first time, every time. You can verify that the update has been installed with the View Installed Updates view in the Windows Update Control Panel applet.