WORK IN PROGRESS
Prometheus is a very nice open-source monitoring system for recording real-time metrics (and providing real-time alerts) in a time-series database for a variety of purposes.
Here we're going to setup Prometheus on a server to monitor a wealth of statistics (such as CPU/memory/disk usage, disk IOps, network traffic, TCP connections , timesync drift, etc.) as well as monitor several endpoints (which could be web applications or services like bitwarden, rocketchat, gitlab, etc.).
We'll also setup a simple dashboard with Grafana, another (great) open-source visualisation and analytics platform that integrates fantastically with (and out-of-the-box) with Prometheus. This will provide a nice dashboard that we'll use to quickly see the status of the endpoints we're monitoring.
An example can be seen at https://mon.jaytaala.com.
Guide
We'll cover the following steps:
Installing and configuring Prometheus (with Docker)
I'm assuming you are running an up-to-date Linux distro and you have Docker installed (if not, you'll need to install Docker).
We only need to run a single command (although it's not a short one) to install and configure our Prometheus setup. To make this a bit easier, I provide my setup files in a git repo you can clone:
https://gitlab.jaytaala.com/docker/prometheus.git
git clone https://gitlab.jaytaala.com/docker/prometheus
The repo contains only two files (which are the only two we need):
- run-container.sh : shell script that will run our docker command to install / configure prometheus.
- prometheus.yml : single configuration file for prometheus;
Before we run the script (which will setup and configure Prometheus) let me point out a few things with these files:
run-container.sh
This script simply creates (and/or runs) our Prometheus container. So what's going on here?
| Line(s) number | Comment |
|---|---|
| 1 | bash script header (tells shell that it's a bash script when it gets executed) |
| 2 | removes / deletes the container. This deletes the already running container (if it exists) before creating a new one. Don't worry(!) our data won't be lost since we're persisting the Prometheus data with a Docker data volume! |
| 3 | main docker run command. The |
| 4 | this data-binds our prometheus.yml config file to |
| 5 | this instructs Docker to create a data volume (for storing our persistent Prometheus data) and call it "prom-data". It's kind of similar to a data-bind but it's managed by Docker and provides many advantages. The actual files (and persistent data) can be found at: |
| 6 | binds port |
Once you've cloned the repo, simply make the script executable (if it isn't already) and run the script with:
chmod +x prometheus.sh ./prometheus.sh
Docker will then download the latest
Installing and configuring node_exporter and blackbox (to monitor endpoints)
Installing and configuring Grafana
References
Related articles