A couple of weeks ago we detailed our action plan for mitigating the Meltdown and Spectre security vulnerabilities and announced that much of the patching was already done. It's going to take a bit longer than originally anticipated.  Read on for the details...

I thought you guys had this taken care of?

So did we.  We expanded our maintenance windows for the last few weeks specifically to expedite the patching of these vulnerabilities and we announced that the majority of our infrastructure was patched.  Then came the big announcement that many of these patches were flawed.  Much of that patching can apparently do as much harm as good swapping security vulnerabilities for stability issues that manifest themselves as random reboots and system crashes.  Basically, Intel gave the hardware vendors some bad code to incorporate into all those BIOS updates and now the vendors need a little time to re-incorporate the new code and test before releasing updates to us.

So what happens now?

Well for starters we have pulled those updates that are known to cause instability from our servers today during another expanded emergency maintenance window.  This is going to be a very short term condition, however, as the vendors are all re-releasing new BIOS patches and the OS vendors will be releasing OS patches again later this month that once again we are being told should fix the problem. Expect additional expanded maintenance windows over the next couple weeks as we apply the corrected BIOS updates and OS patches.  Most major vendors are expecting to release their patches this month.

Is there anything that I need to do?

Not much.  Just be patient with us as we continue to patch - we do this for the safety and security of YOUR data.  Normally our patching takes place between 8AM and 12PM EST on Saturdays but due to the amount of time it takes to do hardware BIOS patches and cluster server upgrades we may be starting a bit earlier for the next few weeks to avoid service interruptions during peak times. As before, stay vigilant and watch this blog for additional updates.