We are currently managing over 2000 virtual machines, hundreds of bare metals, tens of services, and tens of user accounts. You can imagine how difficult it was to add or change existing users (change permission, access, ssh keys, and so on).
The Pains of Locally Managed Users
Previously, our users were deployed only with puppet, which is great, however, searching for users in different git repositories, different branches, wasn’t the right way. Some hosts could have disabled puppet for testing reasons or hardcoded temporary changes, so the user changes weren’t deployed.
Also, adding new users to our services (and we use plenty of them) turned out to be a painful way. Going through all of them and adding new users one by one took a lot of time. Furthermore, we pretty much lost control over who had access to where and with what permission. Moreover, using shared passwords is a big no-go, it is a complete loss of control over access. Of course, most things are behind vpn, firewall, and so on. But you never know…
LDAP Infrastructure
Last summer we decided that we wanted to use some sort of LDAP to handle this. We have chosen OpenLDAP as it is probably the most used LDAP with Linux based systems – currently, our servers are mostly Ubuntu, Debian.
In the current state, we are running a multi-master setup, which consists of 2 master nodes and about 10 read-only slave servers, 2 for each customer. All nodes are in high-availability, using corosync, which functions great. Each used service is connected to a pair of slaves (behind corosync). Slaves are pulling new data from a master node every 15 minutes because, to be honest, we were unable to automatically sync from master to slave when the changes were made.
Implementation
At first, we only created 1 LDAP server which we used for testing. At the beginning it was very difficult to configure the server. Configuration is done through ldapmodify commands, you can’t (well, you shouldn’t) modify config files by hand because sometimes it just stops working. I learned that the hard way.
The first implemented service was ssh, a switch from local auth to LDAP. It was done by using puppet, which edits ssh config files for us. The basic idea is that in sshd config you allow groups which have access to the server – “AllowGroups”. Ssh is friendly with LDAP object of type posixGroup. Second thing is obtaining user’s SSH key, again, in sshd config we can set “AuthorizedKeysCommand” – script, which returns the key based on the user id – UUID. Basic ldap search command.
Then we started switching between different services other than ssh, for example Grafana, Redmine, Poweradmin, Openstack, Git, … Most services are friendly with group of type groupOfNames where you can easily use “memberof” to filter users. However, groupOfNames must be added using ldapmodify command, the same goes for memberof.
When we had this tested, it was time to set up ldap cluster in the right way, which means multi-master with sync to slave the servers.
Frontend
We use a very simple frontend, used to easily manage our users, which is called LDAP Account Manager. You can, of course, make changes in terminal, but that is not user friendly at all.
Monitoring
For every server running ssh ldap auth we monitor its access to a respective ldap slave server.
We also monitor the sync between masters and the master-slave sync using a simple check written in Golang, it has great support for ldap auth.
Summary
At the beginning it was a very hard configuration, until you get used to it. From scratch configuring, our cluster took me over a month of work.
Most of the stuff you want to use or configure can be found on the internet, that, however, doesn’t always work or doesn’t work the way you would expect.
In the end, we have gotten ldap cluster, which is enough for our needs and generally reliable. We have been using it for more than one and a half years without any issues.
Pros
- Reliability
- Security
- Community
- Comfort in finished product
Cons
- Steep learning curve
- Lack of direct support for some services
- Different approach for different services
- The Joys Of LDAP
The original autor: Antonin, Team Lead for part-time/DevOps Delivery Manager, cloudinfrastack
