Prometheus Exporter for Saltstack Highstate metrics
As a new member of the Site Reliability Engineering (SRE) Core Team @ ioki, I had the opportunity to create a custom Prometheus exporter and I wanted to share my experience with the community.
Prometheus is an open-source monitoring and alerting system that is widely used in the DevOps/SRE world and iokiverse. It can scrape metrics from various sources, store them, and provide an easy-to-use interface for querying and visualizing the data.
Before creating the custom Prometheus exporter, the process of collecting saltstack highstate data involved reading the current
salt highstate regularly on each individual machine and writing the data into a file. This file would then be read via the generic node-exporter textfile collector which had to be rolled out onto all machines.
This approach is obviously much harder to extend, change and test than a custom exporter.
Scalability becomes an issue when you have many (!)
Salt Minions trying to invoke the
Salt Master to gather the
These calls occupy more resource and are harder then doing one call from the
Salt Master to each
Salt Minion in batches.
Our custom Prometheus exporter solves the scalability issue by only running on the master node and checking the highstate in batches (default:
10). Hence, allowing to scrape all data from one place. An easily maintainable and testable solution is born, written in a higher-level programming language with a mature ecosystem.
Python enables us to use the already installed salt dependency and provides confidence in the parsing/scraping of information, because we are now reading from the actual
Python implementation of
salt instead of parsing the
salt state.highstate directly.
In summary, the custom Prometheus exporter provides a more scalable, accurate, and maintainable solution for collecting saltstack highstate data and can be easily extended to collect more data. By running it on the
Salt Master node, it reduces the maintenance burden, the resource usage, and makes it easier to collect and aggregate data from all machines managed by the
To make sure the package works as expected, a Pipeline will run
integration tests creating an environment with one
Salt Master and one
Salt Minion via
Github Actions, testing the written metrics and expecting that the exporter only runs a
salt state.highstate instead of an actual run of
salt state.highstate which would apply
There is actually only one step involved in installing the new exporter onto any machine:
pip install prometheus-salt-exporter
Be sure to be on the
Salt Master. Internally we use the LocalClient “which must be run on the same machine as the Salt Master and it must be run using the same user that the Salt Master is running as”.
By default the
9175 port needs to be accessable for
Prometheus. This and many other parts of the exporter can be changed via optional arguments:
Running the code directly
If you wish to run the code without installing from
pypi, follow these steps:
- Clone the repository
- Install the dependencies (prefereably via
pipenvsince we already provide a
- Call the exporter as a module
python -m src.prometheus_salt_exporter.main --help
It is prefered to use
pypi instead of the code because that is going to handle the code and its dependencies automatically 🪄.