Quite recently (5-8 years ago) the Linux system administrator was limited in administration and automation tools. Somewhere it was possible to get by with self-written scripts in bash, Python, Perl, and somewhere an enterprise-level solution from such giants as IBM, Oracle or RedHat was required.
With the development of Open Source, automation in administration began to develop. Ready-made solutions have come to replace self-written scripts and programs. These funds did not appear out of nowhere. These were solutions to automate the existing tasks of any system administrator. Often these are solutions developed according to the KISS principle (an acronym for “Keep it simple, stupid”), which received great prospects for development and distribution.
Of course, 5-10 years ago, centralized administration tools in Windows were better developed than in Linux. This was not unreasonable, since Windows has been widespread among both home users and office / server environments. Microsoft didn’t seem to envision that it would ever compete with Linux in the server segment. But let’s not delve into these stagnant times for Microsoft, when Steve Ballmer was the head of the corporation.
Administration
One of the commonly used administration tools by me and my colleagues before ansible / puppet / chef was cssh (Cluster SSH).
Working with cssh was simple, there was no need to repeat the same actions on each server in turn, it all boiled down to multiplexing the input in the terminal onto a group of SSH connections. This solution, of course, had obvious drawbacks – it was necessary to interact with the administrator, his control. Today automation tools have gone much further in their capabilities and functionality and are often extended by plugins.
One of the most common administration automation tools is Ansible. It allows you to automate almost any task for a system administrator. SSH access to the hosts is enough to work. The site has an overview article about Ansible, how to set it up and work. The project developed independently and was soon bought by RedHat.
Monitoring
After configuring the server and putting it into operation to ensure SLA and not only, you need to pay attention to monitoring.
We often split monitoring into two separate components:
- monitoring with alerts,
- statistics on indicators.
There is no need to set up alerts for all indicators in the system, but collecting system indicators for statistics and further detection of anomalies is useful and convenient.
- Zabbix / Nagios / Icinga – used to receive events by agents and trigger notifications.
- Grafana + InfluxDB – this bundle also includes Chronograf , Kapacitor , Telegraf .
-
- InfluxDB is a time series database that accepts data from various sources as input. Has a wide functionality for working with data.
- Chronograf – web dashboard and control system Influxdb, Kapacitor.
- Kapacitor is an event handler.
- Telegraf is an agent that sends data from remote systems.
- Grafana is a well-known system for building dashboards / charts.
Also in the monitoring category, you should pay attention to Netdata . It is both an agent and a real-time monitoring system with a dashboard for viewing statistics, pre-configured charts and triggers. After installation, it remains to configure only the notification methods indicating the transmission channel.
Netdata is a “Swiss knife” in the monitoring system, has a wide functionality, support for modules in Python, Go and more.
Centralized package management
We used RedHat Spacewalk as a centralized management of the software suite and its updates .
With the release of RHEL 7, RedHat underwent an update, as a result of which they replaced Spacewalk with a RedHat Satellite, which in many ways resembles Foreman. To migrate to the RedHat Satellite, it would be necessary to reconfigure the systems already working with RedHat Spacewalk, debug a new solution (RedHat Satellite) and almost guaranteed to deal with new problems.
Often in government agencies and enterprises, RedHat solutions were widely used. And then, from the background, Oracle comes out, which was engaged in the development of Oracle Enterprise Linux / Oracle Unbreakable Enterprise Kernel distributions.
These are products based on the RedHat Enterprise Linux codebase, but with a large number of modifications for their own needs, including the creation and support of an environment for, probably, the main product of Oracle – Oracle Databases.
But, unlike RedHat, the distribution kit of which could be downloaded only by subscription (licensing), not to mention updates, Oracle provides it for free. It was Oracle who undertook to support and continue the development of Oracle Spacewalk, which was soon updated in Oracle Linux Manager.
Logging
With the increase in the number of hosts served, there is a need for a centralized collection of logs. Ready-made Open Source solutions that we use are perfect for these tasks:
Agents – rsyslog , syslog-ng (available on most distributions)
Log aggregators:
- Loki (Prometheus),
- Graylog2,
- Logstash,
- ELK.
With a small number of administered hosts, syslog-ng can be used as a centralized storage for log files, which will receive messages on UDP port 514. He knows how to sort messages into directories depending on the source (FQDN / IP-address), service, date, etc.
Backup
For backups, the choice is straightforward. Somewhere rsync + tar (synchronization and compression) is enough, but somewhere Bacula / Bareos is required .
There are quite good proprietary solutions in backup , for example, ” Veeam backup “. If you are using VMWare virtualization, then Veeam simplifies backups and provides support.
Also in our solutions we use a backup for the / etc directory – etckeeper . It allows you to automate the saving of the contents of the / etc directory to a version control system (VCS) repository, monitors when your package manager saves changes to / etc when installing or updating packages.
Putting / etc under version control is now considered best practice in the industry. The advantage of etckeeper is that it makes the process as painless as possible and convenient. If there are uncommitted changes, etckeeper will save them daily, unless disabled, and push them to the centralized repository.
Sometimes it happens that a software package prepared for installation / update may overwrite existing files in / etc, because some customers use third party services / packages. Etckeeper is able to fix the configuration before and after installing packages, which greatly facilitates the work.
Postscript
The choice of any tool depends on the tasks at hand. If you have not found a ready-made solution for your tasks, you can always start implementing it yourself. There will certainly be those who will support you and join the development of your solution. Perhaps it is it that will solve problems better than others.
