| Status | ||||
|---|---|---|---|---|
|
Prometheus is a very nice open-source monitoring system for recording real-time metrics (and providing real-time alerts) in a time-series database for a variety of purposes.
...
An example can be seen at https://mon.jaytaala.com.
A bit about Blackbox Exporter, Node_Exporter and Grafana...
| Expand |
|---|
Blackbox ExporterBlackbox exporter is a prometheus exporter which can probe endpoints via http/https, icmp, etc. It's extremely simple to setup and can be used to monitor the status of various endpoints (which could be web applications, REST endpoints, etc.). We've already setup Node_Exporternode_exporter is a prometheus exporter which monitors hardware and 'nix OS metrics. It runs on a port exposed to prometheus and prometheus can then query it and get a (large) ranger of metrics for whatever machine is running node_exporter. GrafanaGrafana is a great platform for visualising data and metrics from large data sets. It can connect with a very large number of data sources and has native (built-in) prometheus support, which makes it extremely easy to integrate prometheus and provides an attractive and versatile front-end to view various prometheus metrics. |
Guide
We'll cover the following steps:
| Table of Contents | ||
|---|---|---|
|
Installing and configuring Prometheus
...
, Blackbox exporter, and Grafana with Docker
...
-Compose
I'm assuming you are running an up-to-date Linux distro and you have Docker and Docker Compose installed (if not, you'll need to install Docker and then Docker Compose).
We Thanks to Docker and Docker Compose, we'll only need to run a single command (although it's not a short one) to install and configure our Prometheus setup. To make this a bit easier, I provide my setup files in a git repo you can clone: to deploy Prometheus, an exporter (which can query endpoints), and Grafana. Let's start by cloning our setup files from my git repo:
https://gitlab.jaytaala.com/docker/prometheus.gitdocker-compose-prometheus-grafana
| Code Block | ||
|---|---|---|
| ||
git clone https://gitlab.jaytaala.com/docker/docker-compose-prometheus-grafana.git |
The repo contains only two files (which are the only two we need):
- run-container.sh : shell script that will run our docker command to install / configure prometheus.
- prometheus.yml : single configuration file for prometheus;
...
several files:
- blackbox-exporter.yml : configuration file for blackbox-exporter
- dashboard-blackbox.json : blackbox exporter dashboard I've customised see Adding monitoring dashboards;
- dashboard-node_exporter.json : node_exporter dashboard I've customised see Adding monitoring dashboards;
- docker-compose.yml : docker compose config file
- grafana.ini : grafana configuration file
- prometheus.yml : prometheus configuration file
Once you've cloned the repo, simply make the script executable (if it isn't already) and run the script withrun (from within the cloned folder):
| Code Block | ||
|---|---|---|
| ||
chmod +x run-container.sh
./run-container.sh |
Docker will then download the latest official Prometheus Docker image, create and configure the container and then run said container.
docker-compose up -d |
If things worked as they should you should be able to visit the basic Prometheus Grafana web interface at http:\\<INTERNAL-SERVER-IP>:100903000.
| Info |
|---|
Replace <INTERNAL-SERVER-IP> with the actual internal IP address of your server, e.g. 192.168.1.x, 10.0.0.x (or whatever it is). |
That's it! You will now have Prometheus running on port 10090.
| Info |
|---|
Note: port 10090 is not the default Prometheus port (which is 9090) - I have another service using that port (which I didn't want to change). See the Explanation of run-container.sh section for notes on changing the port. |
...
This script simply creates (and/or runs) our Prometheus container. So what's going on here?
| View Git file | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
...
removes / deletes the container. This deletes the already running container (if it exists) before creating a new one.
| Tip |
|---|
Don't worry(!) our data won't be lost since we're persisting the Prometheus data with a Docker data volume! |
that wasn't so bad was it?
Troubleshooting
| Expand | ||
|---|---|---|
If it fails it's most likely that port 3000 on the host is taken - in which case simply change the "ports" first value in docker-compose.yml to another port. E.g. for me I used:
|
Optional reading (explanation of configuration files)
...
main docker run command.
The -d flag means "detached" (i.e. run it detached or in the background) and the --name=prom names the container that will be created "prom".
...
this data-binds our prometheus.yml config file to /etc/prometheus/prometheus.yml in the created Docker container. I.e. prometheus running in the container will use this file for it's configuration.
...
this instructs Docker to create a data volume (for storing our persistent Prometheus data) and call it "prom-data".
It's kind of similar to a data-bind but it's managed by Docker and provides many advantages.
The actual files (and persistent data) can be found at:/var/lib/docker/volumes/prom-data/_data
...
binds port 10090 on the host to post 10090 on the container (in other words exposes the prometheus port on the host).
| Info |
|---|
Note: this is not the default port for Prometheus (default port is 9090). Change both of these to whatever port you would like to bind Prometheus to. Also see line number 9 which would also need to match the port entered here. You'll also need to modify the included |
...
| Expand |
|---|
|
...
Explanation of prometheus.ymlThis file is the main Prometheus.yml scrape configuration file. It defines things like how often Prometheus should scrape (query) data and from where (and how) it should scrape this data.
|
...
|
...
|
...
|
...
|
...
Note that 10.0.0.137:9115 refers to the internal IP address and port of the machine on which blackbox is running on. We'll be setting up blackbox on a docker container and then exposing port 9115 to the host.
This is NOT localhost (localhost in this context is the Prometheus Docker container).
This file is the main grafana config file for our setup. It defines things server related settings, as well as settings we'll use to enable public viewing access.
|
Updating Prometheus with Docker
Updating with Docker is straight-forward. We just need to pull the (latest) image and then re-run our run-container.sh script.
| Code Block | ||
|---|---|---|
| ||
| docker pull prom/prometheus
./run-container.sh |
Installing and configuring node_exporter (to monitor server stats)
node_exporter is a prometheus exporter which monitors hardware and 'nix OS metrics. It runs on a port exposed to prometheus and prometheus can then query it and get a (large) ranger of metrics for whatever machine is running node_exporter.
By default node_exporter enables a large number of "collectors" (modules which collect certain information from the machine). See here for a list of collectors enabled by default (and what info they collect).
...
Installing node_exporter can be done by downloading a recent version version, untar'ing and executing. We're going to be doing an extra step here to manage node_exporter with systemd (so it starts on server boot etc.).
...
| Code Block |
|---|
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
ExecStart=/opt/node_exporter/node_exporter --web.listen-address=:10091
[Install]
WantedBy=multi-user.target |
...
| View Git file | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Installing and configuring blackbox (to monitor endpoints)
Blackbox exporter is a prometheus exporter which can probe endpoints via http/https, icmp, etc. It's extremely simple to setup and can be used to monitor the status of various endpoints (which could be web applications, REST endpoints, etc.).
We'll be using the a docker image to install and configure Blackbox exporter.
Start by cloning https://gitlab.jaytaala.com/docker/blackbox:
| Code Block | ||
|---|---|---|
| ||
git clone https://gitlab.jaytaala.com/docker/blackbox |
Similar to our prometheus docker image install, the repo contains only two files (which are the only two we need):
- run-container.sh : shell script that will run our docker command to install / configure blackbox exporter;
- config.yml : single configuration file for for blackbox exporter.
To setup and run our blackbox exporter simply make the run-container.sh executable and run it:
| Code Block | ||
|---|---|---|
| ||
chmod +x run-container.sh
./run-container.sh |
This will start blackbox-exporter on the default port (9115). The config.yml file is similar to the default blackbox-exporter config.yml, but set the preferred ip protocol to ipv4 (by default it uses ipv6).
We've already setup blackbox-exporter in our prometheus configuration file (see lines 36 to 57).
| View Git file | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| Info |
|---|
Note: the internal ip address on line 57 must be the actual ip address of our host (not Add any other endpoints in the |
Installing and configuring Grafana
Grafana is a great
Finally, we install and setup grafana
Setting up Grafana data sources and dashboards
Adding Prometheus as a data source to Grafana
Once you have your docker containers running, access port 3000 on the server with a browser (e.g. visit http://192.168.0.1:3000 or whatever your internal address is for your server).
| Info |
|---|
For external network access you should (shall!) secure client-server comms. See Apache reverse-proxy SSL to multiple server applications for a nice(?) way to do this for our setup here. |
Grafana's login screen should appear - if you're the first user to login after creating this container then just enter any new credentials.
Once actually logged in, click on the "add data source" and select "Prometheus":
You can leave most setting as default and simply enter the docker-compose defined prom hostname with port 9090 for the Prometheus instance we have setup:
| Info |
|---|
|
| Anchor | ||||
|---|---|---|---|---|
|
One of the great things with Grafana is that it is extremely easy to use (and publish) any dashboards that others have created. There's many (many) dashboards for different systems and many for prometheus and it's many exporters (like node_exporter, blackbox, etc.).
At mon.jaytaala.com I've used these dashboard (which I've then customised further to suit my needs):
- https://grafana.com/grafana/dashboards/7587 (for viewing blackbox http endpoint statuses);
- https://grafana.com/grafana/dashboards/1860 (for viewing server system stats);
You can see many other dasboards for grafana/prometheus here.
It's incredibly simple to add a dashboard to Grafana. Select the dashboards icon from the left menu and then select "Manage:
Now click the "import" button:
Either copy the link (e.g. https://grafana.com/grafana/dashboards/7587) or simply note the dashboard number (7587). Enter this into the "Grafana.com Dashboard" input and press the tab key:
| Tip |
|---|
You can also add dashboard via by uploading a dashboard .json. You'll note that I have two dashboard .json files in the repository - you can upload these by clicking the "Upload .json file" button. |
Grafana will grab the dashboard and after a second or so will fetch the dashboard ready for importing. Enter a few options (see arrows) and then click "Import":
References
...
- https://prometheus.io/
- https://github.com/prometheus/node_exporter
- https://prometheus.io/download/#node_exporter
- https://github.com/prometheus/blackbox_exporter
- https://grafana.com/
...