Patching Systems - An Evaluation

by Gregg Berkholtz
GB Computers

October 6, 2003

Last week, Ray Robert said that he spent "the morning" patching his Linux systems and "much of yesterday" patching his Unix systems. He concluded with "Standardizing on one system is looking pretty good to me." He didn't say which system he would choose. How do the major operating systems compare in terms of patching?

Between my employer systems, my personal systems, and the systems of a local consulting firm, I manage (including patching) around 250 Win32 desktops (NT4 and 2K), about 40 Linux, 15 Netware, 3 Sparc Solaris, and 3 SCO OpenServer systems. In general, I've found the systems to be patchable in this order of ease and stability:

Linux

The Debian Linux distribution is by far the best for patching. Security issues are usually fixed within hours and they only patch the existing production code. They don't force you to upgrade to the latest version of the package, which sometimes brings in bugs, incompatibilities, or other quirks I don't want to deal with right then.

The RedHat distribution running with APT (not RPM) is second, because it takes a bit longer (usually by a few hours) to make the patches available.

For both the Debian and RedHat systems, I use a suite of tools known as "APT". The apt tools are native to Debian but were ported to Red Hat by Conectiva http://freshrpms.net/apt. Apt is a tool that automatically determines interdependencies between various programs/packages, and automatically downloads and installs them for you. You can either have APT download and compile the source or you can just have it install the pre-built binary packages.

In what I estimate is 95% of the patches, it's just a matter of running these two commands:

  apt-get update
  apt-get upgrade

The first command updates the local database, so the APT tools know what's available for download and is aware of package/program interdependencies. The second program actually does the work.

I rarely supervise the work. It runs unattended. In seven years, I've never had a significant problem though I do double-check the more critical systems. For the recent OpenSSH/OpenSSL issue, all my Linux servers were patched in 15 to 20 minutes without disrupting production.

In addition to using APT for patching systems, it can install and uninstall software. I've used it to perform major OS upgrades (ie: from Debian 2.2 to 3.0), which included things such as a major library change, as well as major kernel upgrades, and updates to nearly every binary on the systems. Major OS upgrades are a different animal than just a patch installation, and I was genuinely surprised when nothing went wrong - despite all the warnings.

Solaris

I use a tool known as pkg-get, which is modeled after the Linux APT tools and behaves in very much the same way. For Solaris 7, 8, and 9 there's also the Solaris Patch Manager that seems to work.

Windows

We use Novell's ZENWorks to manage patches. Patch revision management is cumbersome and labor intensive even though our environment is fairly standardized. Frequently, Microsoft's patch tools say the system is patched but the actual system files have not yet been replaced. Sometimes some files are replaced but others are not. It's a convoluted mess and consumes much of my time.

Microsoft's products are really in need of a decent, enterprise-manageable patch management process. One that is truly integrated into the OS, and the standard software install routines. One that keeps my systems inventories internal/private to us, and allows for rapid, stable, and reliable forced deployment of patches.

Now there's where Microsoft could "innovate". Instead of trying to stomp-out competitors in other areas, how about catching up to the rest of us? Eh, now I'm dreaming.

Novell

For the NetWare/GroupWise world, patch management is often worse than Windows. Their patch bundles frequently leave a server unbootable, or otherwise unusable. Novell has a long history of only documenting two-thirds of the actual changes included in a patch. Novell techs have admitted this to me, blaming it on a "last minute addition". Uninstalling patches on NetWare is a joke. Windows patches uninstall better.

As for security updates, I've found myself stumbling over security vulnerabilities that are patched in an undocumented manner in major patch releases; no notes with the major patch and no separate security vulnerability announcement. Other times, the only way to install a security patch is to install the whole OS service pack. There seems to be little concept of a hot-fix which resolves a specific issue. Ok, now I'm ranting.

I am aware of no APT-like tools for NetWare either.

We're replacing our SCO servers
as fast as we can.

SCO OpenServer

As best as I know, there are no APT-like tools for SCO, and given SCO's inappropriate actions against the OSS community, development of something seems slim. SCO has sometimes taken longer than six months to announce and release patches for crucial software. OpenServer is so finicky and unstable when it comes to patching, it's hardly surprising automated tools are not available. We're replacing our SCO servers as fast as we can.

Maybe my standards are too high.

Back