WORK IN PROGRESS
Prometheus is a very nice open-source monitoring system for recording real-time metrics (and providing real-time alerts) in a time-series database for a variety of purposes.
Here we're going to setup Prometheus on a server to monitor a wealth of statistics (such as CPU/memory/disk usage, disk IOps, network traffic, TCP connections , timesync drift, etc.) as well as monitor several endpoints (which could be web applications or services like bitwarden, rocketchat, gitlab, etc.).
We'll also setup a simple dashboard with Grafana, another (great) open-source visualisation and analytics platform that integrates fantastically with (and out-of-the-box) with Prometheus. This will provide a nice dashboard that we'll use to quickly see the status of the endpoints we're monitoring.
An example can be seen at https://mon.jaytaala.com.
Guide
We'll cover the following steps:
Installing and configuring Prometheus (with Docker)
I'm assuming you are running an up-to-date Linux distro and you have Docker installed (if not, you'll need to install Docker).
We only need to run a single command (although it's not a short one) to install and configure our Prometheus setup. To make this a bit easier, I provide my setup files in a git repo you can clone:
https://gitlab.jaytaala.com/docker/prometheus.git
git clone https://gitlab.jaytaala.com/docker/prometheus
The repo contains only two files (which are the only two we need):
- run-container.sh : shell script that will run our docker command to install / configure prometheus.
- prometheus.yml : single configuration file for prometheus;
See the Explanation of run-container.sh section for an explanation and several notes for these files.
Once you've cloned the repo, simply make the script executable (if it isn't already) and run the script with:
chmod +x prometheus.sh ./prometheus.sh
Docker will then download the latest official Prometheus Docker image, create and configure the container and then run said container.
If things worked as they should you should be able to visit the basic Prometheus web interface at http:\\<INTERNAL-SERVER-IP>:10090.
Replace <INTERNAL-SERVER-IP> with the actual internal IP address of your server, e.g. 192.168.1.x, 10.0.0.x (or whatever it is).
That's it! You will now have Prometheus running on port 10090.
Note: port 10090 is not the default Prometheus port (which is 9090) - I have another service using that port (which I didn't want to change). See the Explanation of run-container.sh section for notes on changing the port.
(Optional Read) Explanation of run-container.sh
This script simply creates (and/or runs) our Prometheus container. So what's going on here?
(Optional Read) Explanation of prometheus.yml
This file is the main Prometheus.yml scrape configuration file. It defines things like how often Prometheus should scrape (query) data and from where (and how) it should scrape this data.
Updating Prometheus with Docker
Updating with Docker is straight-forward. We just need to pull the (latest) image and then re-run our run-container.sh
script.
docker pull prom/prometheus ./run-container.sh
Installing and configuring node_exporter (to monitor server stats)
Installing and configuring blackbox (to monitor endpoints)
Installing and configuring Grafana
References
Related articles