James Massardo - Geek of all things Technology

Patch all the things!

|

Summary

Greetings! I recently did a customer call/demo around using Chef to patch Windows systems and I thought this would make a great post. However, I’m going to change one thing, we’re going review patching as a fundamental process and cover more than Windows.

Objectives

What are our objectives in patching, e.g. why do we do this seemingly futile task? What business value do we provide? There’s really only a couple things IT can do for a business; things that allow the business to move faster and things that reduce risk to the business. In the case of patching, we’re really trying to reduce the risk to the business by minimizing the possible attack surfaces. According to most security firms, roughly 80% of all data breaches occurring the past few years would have been prevented if those businesses had an effective patching process.

Ok, cool, so our goal is to reduce risk; however, making low level code changes on basically everything in a relatively short time sounds pretty risky too, right? In the post DevOps world, we wouldn’t patch. When new updates are available, they are shipped through the pipeline like every other change. Ephemeral nodes are provisioned, integration tests run, new OS images prepped, new infrastructure provisioned, etc.

I can read your mind at this point; “We aren’t all start-ups, some of us still have a lot of legacy systems to manage.” And you’d be totally correct. Almost all enterprises (and a lot of smaller businesses) have legacy systems that are critical to the business and will be around for the forseeable future. Having said that, this doesn’t mean that you can’t have a well-understood, mostly automated patching process.

Basic tools

Repositories

Below are some notes on each major platform. The main thing to remember is the repository tool is the source for our patches. It’s also the main control point for what’s available in our environment. By default, we want to automatically synchronize with the vendor periodically. We then black list any patch or update that we can’t use (or busted patches that break stuff).

Essentially, we want to create one or more local repositories for all of the available packages for each platforms/distribution we want to support. The total number required will vary depending on your network topology, number of supported platforms/distributions, etc. Patching can be extremely network intensive. If you have a large number of systems or multiple systems across one ore more WAN links, plan accordingly and don’t DDoS yourself. I can’t emphasize this enough, if you don’t take load (network, OS, hypervisor, etc.) into account, you will cause severe, possibly even catastrophic problems for your business.

Now that the gloom and doom warnings are out of the way, let’s look at some OS specific info.

Linux

Each Linux distribution has it’s own repository system. For RHEL based systems, we use Yum, and for Debian based systems, we use Apt.

RHEL has some licensing implications for running local repositories. Talk to your Red Hat rep for more info.

MacOS

Until recently, macOS Server had a Software Update Services component. Some folks are using the new caching service but it’s not quite the same.

Windows

Windows Server has an Update Services role. WSUS can provide local source for the majority of Microsoft’s main products. WSUS also has a replica mode for supporting multiple physical locations or very large environments.

Windows users that are running newer versions of Server and Client can also take advantage of the Branch Cache features. This allows clients on a single subnet to share content and drastically reduce the WAN utilization without needing downstream servers.

Configuration

I’m going to use Chef for my examples (full disclosure, I work at Chef), but these principles will work with any configuration management platform. The main thing to keep in mind is the CM tool doesn’t do the actual patching. This is a pretty common misconception. Yes, it’s entirely possible to use a CM tool to deliver the actual patch payload and oversee the execution, but why would you want to do all that extra work? This is about making our lives better so let’s use existing tools and functionality and use CM to control the operation.

Orchestration

Orchestration means different things to different people. To me, orchestration is the process of managing activities in a controlled behavior. This is different from CM in that CM is about making a system or object act a certain way whereas orchestration is about doing multiple things in a certain order or taking action based on one or more inputs.

You will need an orchestration tool if you have any of the following needs (or similar needs):

  • I need to reboot this system first, then that system for my app to work correctly.
  • I want no more than 25% of my web servers to reboot at any given time.
  • I want to make this DB server primary, then patch the secondary DB server.

If any of these sound familiar, you aren’t alone. These are common problems in the enterprise; however, there’s no common solution. With the magnitude of various applications, systems, and platforms out there, there’s no way to provide prescriptive guidance for this topic. A lot of folks use a outside system as a clearing house for nodes to log their state then check the state with a cookbook on all the nodes to make decisions.

In regard to patching, if I have a group of nodes that has a specific boot order, I tend to stagger their patch windows so the first nodes patch and reboot, then the second group, and so on. I may also patch them in one window and leave them pending reboot, then reach out to them separately and reboot them in the required order.

Process

As you can see, the names of the tools within a group may be different; however, they all function using similar patterns. These similarities are what allow us to build a standardized process regardless of platform.

  • Repositories provide the content and content control
  • Config. mgmt. tools control the configuration of the client (I know… Mr. Obvious…). This is how we assign maintenance windows, reboot behaviors, etc.
  • Orchestration tools handle procedural tasks

We want this to be as automated as possible so everything will be set to run on a scheduled basis. The only time it requires manual intervention is when there’s a patch we need to exclude. In the examples below, we’ll configure the repositories to sync nightly and we’ll configure the clients to check on Sunday mornings at 2AM. I can hear what you are thinking again, you’re saying “We have a bunch of systems, we can’t reboot all of them at once!” And you’d be right. Even if you don’t have a large fleet, you still don’t want to patch all the things at once.

In reality, you want at least one test system for each application, service, or role in your environment. Preferably, you want complete replicas of production (although smaller scale) for testing. Patches should be shipped and tested just like any other code change. Most enterprises have some process similar to Dev -> QA -> Stage -> Prod for shipping code changes so patching should follow that same pattern. Remember, the earlier we discover a problem, the easier and cheaper it is to resolve.

Technical Setup

Below are sample instructions for building the required components. These are not 100% production ready and only serve as examples on where to start. Each company has it’s own flavor of doing things so it’s not possible to account for all the variations.

Server Side

First thing we need is to set up our repositories. We’ll set up a basic mirror for CentOS 7 and Ubuntu 16.04, then set up a Windows WSUS Server.

APT Repository

# metadata.rb
# attributes/default.rb
# recipes/default.rb

Yum Repository

# metadata.rb
depends 'yum'
# attributes/default.rb
default['yum']['repos']['centos-base'] = 'http://mirror.centos.org/centos/7/os/x86_64'
default['yum']['repos']['centos-updates'] = 'http://mirror.centos.org/centos/7/updates/x86_64'
default['yum']['combined'] = false
# recipes/default.rb
package 'createrepo'
package 'python-setuptools'
package 'httpd'

# https://github.com/ryanuber/pakrat
execute 'easy_install pakrat'

repos = ''
node['yum']['repos'].each do |name, baseurl|
  repos += "--name #{name} --baseurl #{baseurl} "
end

repos += '--combined ' if node['yum']['combined']

directory '/var/www/html/' do
  recursive true
end

#
#
# Convert to a cron resource to schedule nightly sync's
#
#########################################################
execute 'background pakrat repository sync' do
  cwd '/var/www/html/'
  command "pakrat #{repos} --repoversion $(date +%Y-%m-%d)"
  live_stream true
end
#########################################################
#
#
#
#

service 'httpd' do
  action [:start, :enable]
end

WSUS Server

# metadata.rb
depends 'wsus-server'
# attributes/default.rb
default['wsus_server']['synchronize']['timeout'] = 0
default['wsus_server']['subscription']['categories']      = ['Windows Server 2016',]

default['wsus_server']['subscription']['classifications'] = ['Critical Updates',
                                                             'Security Updates']
default['wsus_server']['freeze']['name']                  = 'My Server Group'
# recipes/default.rb
include_recipe 'wsus-server::default'
include_recipe 'wsus-server::freeze'

Client Side

Now that we have repositories, let’s configure our clients to talk to them.

CentOS Client

# metadata.rb
# attributes/default.rb
# recipes/default.rb
cron 'Weekly patching maintenance window' do
  minute '0'
  hour '2'
  weekday '7'
  command 'yum upgrade -y'
  action :create
end

Ubuntu Client

# metadata.rb
# attributes/default.rb
# recipes/default.rb

Windows 2016 Client

# metadata.rb
depends 'wsus-client'
# attributes/default.rb

default['wsus_client']['wsus_server']                              = 'http://wsus-server:8530/'
default['wsus_client']['update_group']                             = 'My Server Group'
default['wsus_client']['automatic_update_behavior']                = :detect
default['wsus_client']['schedule_install_day']                     = :sunday
default['wsus_client']['schedule_install_time']                    = 2
default['wsus_client']['update']['handle_reboot']                  = true
# recipes/default.rb
include_recipe 'wsus-client::configure'

Validation

Now the real question: “Did everything get patched?” How do we answer this question? Apt and Yum have no concept of reporting and the WSUS reports can get unwieldy in a hurry. Enter Inspec. Inspec is an open source auditing framework that allows you to test for various compliance and configuration items including patches. Patching baselines exist for both Linux and Windows. Inspec can run remotely and collect data on target nodes or you can have it report compliance data to Chef Automate for improved visibility and dashboards.

Closing

Congratulations! If you are reading this, then you made it through a ton of information. Hopefully you found at least a couple nuggets that will help you. If you have any questions or feedback, please feel free to contact me: @jamesmassardo

Thanks to @ncerny and @trevorghess for their input and code samples.

Helpful tips

Here are some tidbits that may help you.

  • WSUS memory limit event log error. You will almost 100% hit this at some point. This is a scaling config so the more clients you have, the more memory WSUS will need.
  • Use attributes to feed required data (maint. window, patch time, reboot behavior, etc.) to the node. This allows you to have one main patching cookbook that get’s applied to everything. You can deliver attributes different ways:
    • Environments
    • Roles
    • Role/wrapper cookbooks
  • Remember we are using our CM system to manage the configs, not do the actual patching.

Pipelining Chef with Jenkins

|

Summary

Today, I’m going to detail out how we are setting up our pipelining process for all things Chef for our business unit partners. This process uses a combination of Chef, Jenkins, GitHub, a custom Jenkins Shared Library, and some custom developed utilities.

Problem

There are a couple problems that started us down this path:

  • We run a single Chef server/Chef org containing all of our systems.
    • This means that we needed a way to allow users to execute changes against the Chef server without allowing knife access.
    • This also means that without automation of some sort, someone from our team would have to execute any change required, making us a serious bottleneck.
  • Knife has no safety net.
    • It’s very easy to make simple mistakes that can cause massive problems.
    • Chef Server/Knife doesn’t have granular RBAC so allowing knife would also allow users the ability to change objects that may not belong to them. We’re all adults and professionals and wouldn’t do anything maliciously to negatively affect another group; however, mistakes happen.
  • Allowing multiple groups to make changes without a consistent process creates numerous problems by allowing variations. This makes performing upgrades and validation testing exponentially harder.
  • Another problem is ensuring that global cookbooks and attributes are applied to all systems. OS level cookbooks that set basic settings for the environment and other things like audit cookbooks used by the internal security teams must be applied to everything to ensure a consistent experience and process.

Solution

I won’t lie to you, this is still very much a work in progress but this is still a significant step forward compared to manual processes.

Let’s look at the components used.

Component Purpose
Chef CM Tool (I know… Obvious, right?)
Jenkins Automation Server
GitHub Source code repository
Shared Library This library is essentially a standard Jenkins pipeline. We elected to use a shared library to simplify the process for when we needed to make changes. We change the library and all the repositories automatically execute the change the next time they are run.
Utilities These utilities handle some of the file merging, interaction with the Chef server, and provide some of the safety nets.
This project also contains the global_envs folder

For deeper detail about each of these components, view their websites or project README’s.

Deployment Processes

Before we get into the setup, I think it’s good to provide some details about our methodology about object deployments. There are two main types of deployments for us:

  • Global Deployments - cookbooks, environment version pins, audit profiles, etc. that are deploy by our Internal IT team.
    • We use the global_envs folder to manage cookbook version pins per environment. This allows us to systematically roll out cookbook changes across the fleet. The environment file is only a section of a standard file. It only contains the cookbook_versions, default_attributes, and override_attributes sections. There is a step in the Jenkinsfile for this project that handles updating the version pins for all the global changes.
  • BU Deployments - cookbooks, environment version pins, and data bags that the BU can manage.
    • Cookbooks
      • Each cookbook has a dedicated Jenkinsfile within the root of the repository that reference the promoteCookbook('env_name') method of the Shared library.
    • Data bags
      • All data bags (including chef-vault bags) are stored in the BU/environment repository. There should only be one environment repository per BU/env. This process follows the same pattern as ChefDK. Create the data bag locally just as you would except instead of using knife data bag from file, the data bag folder is added to the repository and committed to GitHub.
    • Environment
      • This is a standard Chef environment file. The automation utilities merge the matching environment file from the global_envs folder with the BU’s environments/name.json file. The pipeline then validates that all of the cookbooks and named versions exist on the Chef server then it uploads it.
    • Profiles
      • Each profile has a dedicated Jenkinsfile within the root of the repository that reference the uploadProfile() method of the Shared library.

The astute reader will no doubt notice the distinct absence of information about run lists. This is a deliberate move. We decided that using the concept of role cookbooks was the best pattern for our company. When we onboard a new BU partner, we establish two role cookbooks.

  • Environment cookbook - This cookbook is managed by our internal IT staff. It contains the required global cookbooks such as our audit profiles.
  • BU Role cookbook - This cookbook is managed by the BU. They add additional cookbooks as needed to meet their needs. They use attributes and guards to control where the code is executed.

Setup

These steps assume you have a working Chef Server and a working Jenkins Server. If you are new to Chef, I suggest heading over to Learn Chef Rally and checking out some of the modules (I would recommend the Infrastructure Automation). If you are new to Jenkins, I would recommend checking out the official Jenkins documentation here: Installing Jenkins.

  • Let’s set up the automation utilities first.
    • Fork the utilities project fork image
    • Create a pipeline in Jenkins/BlueOcean new pipeline image
    • Select GitHub (I’m sure this would work if the code was stored somewhere else, we just happen to use GitHub for our stuff.) new pipeline image
    • Select the appropriate organization new pipeline image
    • Select the repository (In our case, we use the Chef-Pipeline-Automation repo.) new pipeline image
  • Ok cool! Now we have our utilities available in the /var/chef_automation/ folder.
  • Next, let’s move on to the Shared Library.
    • Fork the project
    • Follow the steps outlined in the project’s README file.
    • I will leave you with a couple notes:
      • Any changes to your fork of the library’s project will be immediately available on the Jenkins server so be careful with any changes or enhancements that you make.
      • This library is essentially a collection of pipelines so if a new function is needed, start by creating a regular Jenkinsfile in a test repo and work out the steps first, then create a new verbNoun.groovy in the /vars/ folder.
  • Now that we have the Jenkins stuff set up, let’s set up the chef-repo.
    • Install ChefDK on the Jenkins server
    • Create a chef_repo folder and place the knife.rb and the user.pem files in the Jenkins home directory
    $ cd /var/lib/jenkins
    $ mkdir -p chef_repo/.chef
    $ cp /path/to/my/knife.rb /var/lib/jenkins/chef_repo/.chef/knife.rb
    $ cp /path/to/my/user.pem /var/lib/jenkins/chef_repo/.chef/user.pem
    $
    $ tree /var/jenkins
    chef_repo
    ├── .chef
    │   ├── knife.rb
    │   └── client.pem
    

Thanks for sticking with me thus far. I’m happy to tell you that all of the ground work is done and the next two steps are to set up the BU’s main repo and their first cookbook and profile repo.

  • BU/Environment repository
    • There needs to be one repository for each environment on the Chef Server.
    • The repository follows this structure:
      env_repo
      ├── environments
      │   ├── env_name.json
      ├── data_bags
      │   ├── admin_users
      │   │   ├── admin1.json
      │   │   ├── admin2.json
      │   │   └── admin3.json
      │   ├── limited_users
      │   │   ├── user1.json
      │   │   ├── user2.json
      │   │   └── user3.json
      │   └── op_users
      │       ├── op1.json
      │       ├── op2.json
      │       └── op3.json
      ├── Jenkinsfile
      └── README.md
      
      • The Jenkinsfile for the environments repository should contain the following:
      // Jenkinsfile
      @Library('chef_utilities') _
      
      updateEnvironment('chef_env_name')
      
  • Cookbook repositories
    • Chef best practices says that each cookbook should be stored in its own repository so that’s how we’ll do it.
    • Create cookbooks as normal chef generate cookbook cookbooks/my_new_cookbook
    • Place a Jenkinsfile in the root of the repository with the contents below:
    // Jenkinsfile
    @Library('chef_utilities') _
    
    promoteCookbook('chef_env_name')
    
    • Once the Jenkinsfile is created, create a new pipeline in Jenkins for each cookbook. If the permissions are set correctly in Jenkins, this task can be delegated to the BU further reducing the necessity for outside intervention.
  • Last but certainly not least, Inspec profiles. Profiles follow the standard processes just like cookbooks with the only difference being the function called in the Jenkinsfile.
// Jenkinsfile
@Library('chef_utilities') _

uploadProfile()

That’s it. Easy peasy, right? I know it feels like a lot of information and a lot of moving pieces and to a certain degree, it is. However, taking the time to set up this type of pipelining greatly improves operational efficiency and drastically reduces the likelihood of a simple mistake breaking production.

Thank you for taking the time to read all of this information. The Automation Utilities and the Shared Library are both open source so feel free to contribute. If you do make enhancements or have questions, please drop me a email, hit me on Twitter, open an issue, or submit a PR on GitHub.

TravisCI Integration

|

Let the continuous integration begin! It took a little tinkering (mostly around getting the FTP upload to work), but I have TravisCI working with GitHub. I’m planning to detail out the process in a future post.

Basic Jeykll!

|

I guess I should blog some of the stuff I’m tinkering with so I don’t forget what I’m learning. I’m going to start at the base and log a few bits and commands about getting started with Jekyll.

First off, you need to install Jekyll. I’m currently using a Mac and I have the full build of Ruby installed for some Chef tinkering so I was able to follow the Jekyll Quickstart. I did have to change the install path as described on the Jekyll Troubleshooting page.

Ok, hopefully you got it installed so let’s actually do something with it.

# Install Jekyll and Bundler gems through RubyGems
~ $ gem install jekyll bundler

# Create a new Jekyll site at ./myblog
~ $ jekyll new myblog

# Change into your new directory
~ $ cd myblog

# Build the site on the preview server
~/myblog $ bundle exec jekyll serve

# Now browse to http://localhost:4000

(Yeah, I know, looks familar, eh? That’s cause I copied it straight from the Quickstart guide… It works, why reinvent the wheel…)

Next is to create a post.

# Create post file using the following format
# YEAR-MONTH-DAY-title.MARKUP 
~/myblog $ touch _posts/2017-06-05-basic-jekyll.md

From there, add your header and start writing. Post Docs

Dear Future Self

|

I met a lot of really awesome people at ChefConf. One guy, named Daniel, and I were joking about blogging stuff and we said we should start a blog named Dear Future Self. Technologists come in contact with so much information, there’s no way anyone can remember it all. A lot of the stuff on this blog is literally so I don’t have to remember everything.

So Dear Future self, please enjoy all this stuff :)