Eliminating root access

Recently, we've upgraded our hardware infrastructure. Instead of migrating all existing machines, we decided to go with the "setup all systems from scratch" strategy. We had a couple of reasons to follow this strategy.

First of all, we wanted to increase the level of control over the services installed and configuration changes made on an individual virtual machine. Secondly, we wanted to eliminate root access to any (virtual) machine. Because over time, more and more people gained root access through "sudo". Each access was granted for a valid reason. For instance when someone needed to monitor/admin a process, or if they had to install some extra packages that other software components relied on. You can imagine that over time, some of our systems had a ton of entries in the sudo-file.

For managing our infrastructure we have been using Puppet to some extent. So, we decided to go all the way. We decided that every package installation, deamon or service configuration change had to be a controlled change. And we decided to use Puppet for this. No admin would be allowed to do any manual installation or configuration at any time.

At the moment, we fully manage all (virtual) machines with Puppet/Hiera. The Puppet configuration consists of a source control system (Stash). To accomplish changes, we have an approval process in place. Each change has to be committed to our source control system where it will be validated and approved, after which it will automatically be deployed.

Looking back at this operation, these are some of the things we learned:

• Controlled changes are great! It helped us several times to look back in our changelog to reproduce why it was changed in the first place, and by whom.
• After having the base system in place, rolling out new more equal systems is really, really easy.
• Puppet has a relatively high learning curve. Especially if combined with Hiera, the 'runtime' model isn't always that obviously.
• Keep Puppet modules simple. Complex modules doing many things at a time are hard to maintain
• It's absolutely vital to have an good testing environment. We have been using vagrant for this purpose.
• Root access is indeed an unnecessary feature.

I would definitely advice everybody to get some kind of system in place where changes are tracked and root access is eliminated or even better: banned. For us Puppet is a great choice, but I'm also aware that there are other great tools on the market.

In the end it's all about getting the job done, deliver quality and be sure to have fun!