Our transition from lower Puppet version to Puppet 5 was (and still is) somewhat tricky. For us – managing more than 2000 hosts with puppet – this task is really time consuming. Every host must be switched manually to make sure no critical changes will apply.
Luckily, there are some steps that can be done to simplify this task, some of which I’m going to explain. This will allow you to switch to higher version of Puppet without losing your precious data, and allow you to use other Puppet 5 features in the future.
Create new Puppet stack
This requires additional resources, but you will be able to switch hosts one by one and you won’t risk any module update messing up with your prod (I’ll explain why in the folowing step). For start, a stack of 3-5 puppetservers is really enough, but as you switch more and more hosts, you will probably want to decomission old puppetservers and add new puppetservers to your new stack. Good idea is to assign new/different domain to your stack.
Create new git branches or git repositories
Creating new git branch or repository dedicated to Puppet 5 hosts only is the main reason for why you created new stack. Make an exact copy of your current repository or create a new “second production” branch, eg. production_puppet5 (that doesn’t look so attractive, but keep in mind that once your upgrade is done, you can rename it back to whatever you want).
Update modules that cause trouble
Now the “fun” part starts. Puppet 5 comes with few changes in code, so there will be some modules that don’t work on Puppet 5. You can easily identify those conflicts by simply installing Puppet 5 puppet-agent package on a host and running puppet. Many of these conflicts can be fixed by refactoring a small piece of your code (mostly a variable type), other will require to update your package. Do this on your development branch/puppetserver and deploy on your new branch/repository when you’re ready. (WARNING: This part is tricky, so be careful not to break any of your already switched hosts.)
Switching hosts one by one
When modules are ready, start switching your hosts. You’ll need to uninstall old puppet package and install updated one. This will probably delete your certificate, so you’ll also need a new one. I advise you to switch hosts per hostgroup, as these hosts share the same code. Don’t forget about hiera. Better be safe than sorry – use –noop flag as a prevention when puppetizing with Puppet 5 for the first time! It will point to changes that are about to happen and might save you from some unpleasant situations. So, as you can see, switching to newer, incompatible version of Puppet isn’t impossible, but requires some time, patience and deliberance. If you’re careful, you have nothing to fear of.
