Prometheus Exporter for Saltstack Highstate metrics

As a new member of the Site Reliability Engineering (SRE) Core Team @ ioki, I had the opportunity to create a custom Prometheus exporter and I wanted to share my experience with the community.

Status Quo

Prometheus is an open-source monitoring and alerting system that is widely used in the DevOps/SRE world and iokiverse. It can scrape metrics from various sources, store them, and provide an easy-to-use interface for querying and visualizing the data.

Before creating the custom Prometheus exporter, the process of collecting saltstack highstate data involved reading the current salt highstate regularly on each individual machine and writing the data into a file. This file would then be read via the generic node-exporter textfile collector which had to be rolled out onto all machines.

This approach is obviously much harder to extend, change and test than a custom exporter. Scalability becomes an issue when you have many (!) Salt Minions trying to invoke the Salt Master to gather the highstate. These calls occupy more resource and are harder then doing one call from the Salt Master to each Salt Minion in batches.

Custom Exporter

Our custom Prometheus exporter solves the scalability issue by only running on the master node and checking the highstate in batches (default: 10). Hence, allowing to scrape all data from one place. An easily maintainable and testable solution is born, written in a higher-level programming language with a mature ecosystem.

Using Python enables us to use the already installed salt dependency and provides confidence in the parsing/scraping of information, because we are now reading from the actual Python implementation of salt instead of parsing the json or text output salt state.highstate directly.

In summary, the custom Prometheus exporter provides a more scalable, accurate, and maintainable solution for collecting saltstack highstate data and can be easily extended to collect more data. By running it on the Salt Master node, it reduces the maintenance burden, the resource usage, and makes it easier to collect and aggregate data from all machines managed by the Salt Master.

To make sure the package works as expected, a Pipeline will run integration tests creating an environment with one Salt Master and one Salt Minion via Github Actions, testing the written metrics and expecting that the exporter only runs a dry-run of salt state.highstate instead of an actual run of salt state.highstate which would apply nonhigh states.

Usage

There is actually only one step involved in installing the new exporter onto any machine:

  1. pip install prometheus-salt-exporter

Be sure to be on the Salt Master. Internally we use the LocalClient “which must be run on the same machine as the Salt Master and it must be run using the same user that the Salt Master is running as”.

By default the 9175 port needs to be accessable for Prometheus. This and many other parts of the exporter can be changed via optional arguments:

usage: prometheus_salt_exporter [-h] [--listen-addr LISTEN_ADDR] [--listen-port LISTEN_PORT] [--highstate-interval HIGHSTATE_INTERVAL] [--wait-on-error-interval WAIT_ON_ERROR_INTERVAL]
               [--batch-size BATCH_SIZE] [--salt-target SALT_TARGET] [--log-level LOG_LEVEL]

options:
-h, --help            show this help message and exit
--listen-addr LISTEN_ADDR
                        Address to bind to (default: 0.0.0.0)
--listen-port LISTEN_PORT
                        Port to bind to (default: 9175)
--highstate-interval HIGHSTATE_INTERVAL
                        Seconds between each highstate test run (default: 300)
--wait-on-error-interval WAIT_ON_ERROR_INTERVAL
                        Seconds to wait when an error occurs (e.g. salt-master not responding in time) (default: 300)
--batch-size BATCH_SIZE
                        Batch size to use in salt (default: 10)
--salt-target SALT_TARGET
                        Salt target to be used (default: *)
--log-level LOG_LEVEL
                        log level (default: 30)

Running the code directly

If you wish to run the code without installing from pypi, follow these steps:

  1. Clone the repository
  2. Install the dependencies (prefereably via pipenv since we already provide a Pipfile)
  3. Call the exporter as a module python -m src.prometheus_salt_exporter.main --help

It is prefered to use pypi instead of the code because that is going to handle the code and its dependencies automatically 🪄.

Important URLs