WORK IN PROGRESS
Prometheus is a very nice open-source monitoring system for recording real-time metrics (and providing real-time alerts) in a time-series database for a variety of purposes.
Here we're going to setup Prometheus on a server to monitor a wealth of statistics (such as CPU/memory/disk usage, disk IOps, network traffic, TCP connections , timesync drift, etc.) as well as monitor several endpoints (which could be web applications or services like bitwarden, rocketchat, gitlab, etc.).
We'll also setup a simple dashboard with Grafana, another (great) open-source visualisation and analytics platform that integrates fantastically with (and out-of-the-box) with Prometheus. This will provide a nice dashboard that we'll use to quickly see the status of the endpoints we're monitoring.
An example can be seen at https://mon.jaytaala.com.
Guide
We'll cover the following steps:
Installing and configuring Prometheus (with Docker)
I'm assuming you are running an up-to-date Linux distro and you have Docker installed (if not, you'll need to install Docker).
We only need to run a single command (although it's not a short one) to install and configure our Prometheus setup. To make this a bit easier, I provide my setup files in a git repo you can clone:
https://gitlab.jaytaala.com/docker/prometheus.git
git clone https://gitlab.jaytaala.com/docker/prometheus
The repo contains only two files (which are the only two we need):
- run-container.sh : shell script that will run our docker command to install / configure prometheus.
- prometheus.yml : single configuration file for prometheus;
See the Explanation of run-container.sh section for an explanation and several notes for these files.
Once you've cloned the repo, simply make the script executable (if it isn't already) and run the script with:
chmod +x run-container.sh ./run-container.sh
Docker will then download the latest official Prometheus Docker image, create and configure the container and then run said container.
If things worked as they should you should be able to visit the basic Prometheus web interface at http:\\<INTERNAL-SERVER-IP>:10090.
Replace <INTERNAL-SERVER-IP> with the actual internal IP address of your server, e.g. 192.168.1.x, 10.0.0.x (or whatever it is).
That's it! You will now have Prometheus running on port 10090.
Note: port 10090 is not the default Prometheus port (which is 9090) - I have another service using that port (which I didn't want to change). See the Explanation of run-container.sh section for notes on changing the port.
(Optional Read) Explanation of run-container.sh
This script simply creates (and/or runs) our Prometheus container. So what's going on here?
Line(s) number | Comment |
---|---|
1 | bash script header (tells shell that it's a bash script when it gets executed) |
2 | removes / deletes the container. This deletes the already running container (if it exists) before creating a new one. Don't worry(!) our data won't be lost since we're persisting the Prometheus data with a Docker data volume! |
3 | main docker run command. The |
4 | this data-binds our prometheus.yml config file to |
5 | this instructs Docker to create a data volume (for storing our persistent Prometheus data) and call it "prom-data". It's kind of similar to a data-bind but it's managed by Docker and provides many advantages. The actual files (and persistent data) can be found at: |
6 | binds port 10090 on the host to post 10090 on the container (in other words exposes the prometheus port on the host). Note: this is not the default port for Prometheus (default port is 9090). Change both of these to whatever port you would like to bind Prometheus to. Also see line number 9 which would also need to match the port entered here. You'll also need to modify the included |
7 | Tells Docker that this container should start/restart whenever Docker starts/restarts (for example on boot etc.). |
8 | Points Docker to the official Prometheus image. Docker will download this image if it hasn't already. |
9-13 | Commands for configuration sent to the Prometheus executable in the container. Most of these are pointing to the various locations were have previous bound with data-binds or data-volumes. |
14 | Disabled. Uncomment to enable the admin-api (so can accept POST queries to modify / delete data etc.). |
(Optional Read) Explanation of prometheus.yml
This file is the main Prometheus.yml scrape configuration file. It defines things like how often Prometheus should scrape (query) data and from where (and how) it should scrape this data.
Line(s) number | Comment |
---|---|
1-5 | Default scraping config for how often Prometheus should scrape. Note, these can be overridden for specific scrape configs (see from line 22). |
23-29 | Prometheus (be default) monitors itself and can report on scrape query duration, samples added to db, data compaction stats etc. Note that |
31-34 | Scrape configutation for node_exporter, an exporter which scrape server machine metrics. We haven't set this up yet but will in the following sections. Note that This is NOT localhost (localhost in this context is the Prometheus Docker container). |
36-57 | Scrape configuration for blackbox, a prober that can probe endpoints over HTTP, HTTPS, DNS, TCP and ICMP. We haven't set this up yet but will in the following sections. Note that This is NOT localhost (localhost in this context is the Prometheus Docker container). |
Updating Prometheus with Docker
Updating with Docker is straight-forward. We just need to pull the (latest) image and then re-run our run-container.sh
script.
docker pull prom/prometheus ./run-container.sh
Installing and configuring node_exporter (to monitor server stats)
node_exporter is a prometheus exporter which monitors hardware and 'nix OS metrics. It runs on a port exposed to prometheus and prometheus can then query it and get a (large) ranger of metrics for whatever machine is running node_exporter.
By default node_exporter enables a large number of "collectors" (modules which collect certain information from the machine). See here for a list of collectors enabled by default (and what info they collect).
node_exporter can be run from a docker container, but it's not recommended since it should be run directly on the host hardware to collect stats.
Installing node_exporter can be done by downloading a recent version version, untar'ing and executing. We're going to be doing an extra step here to manage node_exporter with systemd (so it starts on server boot etc.).
We start with downloading. You can find a link for the latest version at https://prometheus.io/download/#node_exporter. At the time of this writing the latest stable version for linux-amd64 was node_exporter-0.18.1.linux-amd64.tar.gz.
We'll download, untar, and then move it to /opt/node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz tar -xf node_exporter-0.18.1.linux-amd64.tar.gz sudo mv node_exporter-0.18.1.linux-amd64 /opt/node_exporter
With node_exporter installed let's make it easier to manage by creating a systemd service. Create a file in /etc/systemd/system/ with your favourite text editor (I'm using vim here):
sudo vim /etc/systemd/system/node_exporter.service
and paste the following:
[Unit] Description=Node Exporter Wants=network-online.target After=network-online.target [Service] Type=simple ExecStart=/opt/node_exporter --web.listen-address=:10091 [Install] WantedBy=multi-user.target
I'm using a non-default port here (10091) so change it to whatever port you prefer (or have free).
Finally let's enable it (to start on boot) and start the service
sudo systemctl daemon-reload sudo systemctl enable --now node_exporter
We've already setup node_exporter in our prometheus configuration file (see lines 31 to 34).
Installing and configuring blackbox (to monitor endpoints)
Blackbox exporter is a prometheus exporter which can probe endpoints via http/https, icmp, etc. It's extremely simple to setup and can be used to monitor the status of various endpoints (which could be web applications, REST endpoints, etc.).
We'll be using the a docker image to install and configure Blackbox exporter.
Start by cloning https://gitlab.jaytaala.com/docker/blackbox:
git clone https://gitlab.jaytaala.com/docker/blackbox
Similar to our prometheus docker image install, the repo contains only two files (which are the only two we need):
- run-container.sh : shell script that will run our docker command to install / configure blackbox exporter;
- config.yml : single configuration file for for blackbox exporter.
To setup and run our blackbox exporter simply make the run-container.sh executable and run it:
chmod +x run-container.sh ./run-container.sh
This will start blackbox-exporter
on the default port (9115). The config.yml
file is similar to the default blackbox-exporter
config.yml
, but set the preferred ip protocol to ipv4 (by default it uses ipv6).
We've already setup blackbox-exporter
in our prometheus configuration file (see lines 36 to 57).
Note: the internal ip address on line 57 must be the actual ip address of our host (not 127.0.0.1
as from the docker image, 127.0.0.1
is the docker image and not the host machine/server).
Add any other endpoints in the targets
section of the yml.
Installing and configuring Grafana
Grafana is a great
Finally, we install and setup grafana
References
- https://prometheus.io/
- https://github.com/prometheus/node_exporter
- https://prometheus.io/download/#node_exporter
- https://github.com/prometheus/blackbox_exporter
- https://grafana.com/
Related articles