Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

An example can be seen at https://mon.jaytaala.com.

Guide

We'll cover the following steps:

Table of Contents
minLevel3

Installing and configuring Prometheus (with Docker)

I'm assuming you are running an up-to-date Linux distro and you have Docker installed (if not, you'll need to install Docker).

We only need to run a single command (although it's not a short one) to install and configure our Prometheus setup.  To make this a bit easier, I provide my setup files in a git repo you can clone: 

https://gitlab.jaytaala.com/docker/prometheus.git

Code Block
languagebash
git clone https://gitlab.jaytaala.com/docker/prometheus

The repo contains only two files (which are the only two we need):

  • run-container.sh : shell script that will run our docker command to install / configure prometheus.
  • prometheus.yml : single configuration file for prometheus;

See the Explanation of run-container.sh section for an explanation and several notes for these files.

Once you've cloned the repo, simply make the script executable (if it isn't already) and run the script with:

Code Block
languagebash
chmod +x run-container.sh
./run-container.sh

Docker will then download the latest official Prometheus Docker image, create and configure the container and then run said container. 

If things worked as they should you should be able to visit the basic Prometheus web interface at http:\\<INTERNAL-SERVER-IP>:10090.

Info

Replace <INTERNAL-SERVER-IP> with the actual internal IP address of your server, e.g. 192.168.1.x, 10.0.0.x (or whatever it is).

That's it!  You will now have Prometheus running on port 10090.

Info

Note: port 10090 is not the default Prometheus port (which is 9090) - I have another service using that port (which I didn't want to change).  See the Explanation of run-container.sh section for notes on changing the port.

...

This script simply creates (and/or runs) our Prometheus container.  So what's going on here?

View Git file
pathrun-container.sh
repository-id8
languagebash
branchrefs/tags/documented
linenumberstrue

...

removes / deletes the container.  This deletes the already running container (if it exists) before creating a new one.

Tip

Don't worry(!) our data won't be lost since we're persisting the Prometheus data with a Docker data volume!

...

main docker run command.

The -d flag means "detached" (i.e. run it detached or in the background) and the --name=prom names the container that will be created "prom".

...

this data-binds our prometheus.yml config file to /etc/prometheus/prometheus.yml in the created Docker container.  I.e. prometheus running in the container will use this file for it's configuration.

...

this instructs Docker to create a data volume (for storing our persistent Prometheus data) and call it "prom-data".

It's kind of similar to a data-bind but it's managed by Docker and provides many advantages.

The actual files (and persistent data) can be found at:
/var/lib/docker/volumes/prom-data/_data

...

binds port 10090 on the host to post 10090 on the container (in other words exposes the prometheus port on the host).

Info

Note: this is not the default port for Prometheus (default port is 9090).  Change both of these to whatever port you would like to bind Prometheus to.

Also see line number 9 which would also need to match the port entered here.

You'll also need to modify the included prometheus.yml file and replace 10090 to whatever port you want to use.

...

This file is the main Prometheus.yml scrape configuration file.  It defines things like how often Prometheus should scrape (query) data and from where (and how) it should scrape this data.

View Git file
pathprometheus.yml
repository-id8
branchrefs/tags/documented
linenumberstrue

...

Prometheus (be default) monitors itself and can report on scrape query duration, samples added to db, data compaction stats etc.

Info

Note that localhost:10090 refers to the Prometheus docker container and not the server.

...

Scrape configutation for node_exporter, an exporter which scrape server machine metrics.  We haven't set this up yet but will in the following sections.

Info

Note that 10.0.0.137:10091 refers to the internal IP address of the machine you want to monitor (and the port on which node_exporter is running). 

This is NOT localhost (localhost in this context is the Prometheus Docker container).

...

Scrape configuration for blackbox, a prober that can probe endpoints over HTTP, HTTPS, DNS, TCP and ICMP.  We haven't set this up yet but will in the following sections.

Info

Note that 10.0.0.137:9115 refers to the internal IP address and port of the machine on which blackbox is running on.  We'll be setting up blackbox on a docker container and then exposing port 9115 to the host. 

This is NOT localhost (localhost in this context is the Prometheus Docker container).

Updating Prometheus with Docker

Updating with Docker is straight-forward.  We just need to pull the (latest) image and then re-run our run-container.sh script.

Code Block
languagebash
docker pull prom/prometheus
./run-container.sh

Installing and configuring node_exporter (to monitor server stats)

node_exporter is a prometheus exporter which monitors hardware and 'nix OS metrics.  It runs on a port exposed to prometheus and prometheus can then query it and get a (large) ranger of metrics for whatever machine is running node_exporter.

By default node_exporter enables a large number of "collectors" (modules which collect certain information from the machine).  See here for a list of collectors enabled by default (and what info they collect).

node_exporter can be run from a docker container, but it's not recommended since it should be run directly on the host hardware to collect stats.

Installing node_exporter can be done by downloading a recent version version, untar'ing and executing.  We're going to be doing an extra step here to manage node_exporter with systemd (so it starts on server boot etc.).

We start with downloading.  You can find a link for the latest version at https://prometheus.io/download/#node_exporter.  At the time of this writing the latest stable version for linux-amd64 was node_exporter-0.18.1.linux-amd64.tar.gz.

We'll download, untar, and then move it to /opt/node_exporter

Code Block
languagebash
wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
tar -xf node_exporter-0.18.1.linux-amd64.tar.gz
sudo mv node_exporter-0.18.1.linux-amd64 /opt/node_exporter

With node_exporter installed let's make it easier to manage by creating a systemd service.  Create a file in /etc/systemd/system/ with your favourite text editor (I'm using vim here):

Code Block
languagebash
sudo vim /etc/systemd/system/node_exporter.service

and paste the following:

Code Block
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/opt/node_exporter --web.listen-address=:10091

[Install]
WantedBy=multi-user.target
Info

I'm using a non-default port here (10091) so change it to whatever port you prefer (or have free).

Finally let's enable it (to start on boot) and start the service

Code Block
languagebash
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

We've already setup node_exporter in our prometheus configuration file (see lines 31 to 34). 

View Git file
pathprometheus.yml
lastline34
repository-id8
firstline31
branchrefs/tags/documented
linenumberstrue
 

Installing and configuring blackbox (to monitor endpoints)

Blackbox exporter is a prometheus exporter which can probe endpoints via http/https, icmp, etc.  It's extremely simple to setup and can be used to monitor the status of various endpoints (which could be web applications, REST endpoints, etc.).

We'll be using the a docker image to install and configure Blackbox exporter.

Start by cloning https://gitlab.jaytaala.com/docker/blackbox:

Code Block
languagebash
git clone https://gitlab.jaytaala.com/docker/blackbox

Similar to our prometheus docker image install, the repo contains only two files (which are the only two we need):

  • run-container.sh : shell script that will run our docker command to install / configure blackbox exporter;
  • config.yml : single configuration file for for blackbox exporter.

To setup and run our blackbox exporter simply make the run-container.sh executable and run it:

Code Block
languagebash
chmod +x run-container.sh
./run-container.sh

This will start blackbox-exporter on the default port (9115).  The config.yml file is similar to the default blackbox-exporter config.yml, but set the preferred ip protocol to ipv4 (by default it uses ipv6).

We've already setup blackbox-exporter in our prometheus configuration file (see lines 36 to 57). 

View Git file
pathprometheus.yml
lastline57
repository-id8
firstline36
branchrefs/tags/documented
linenumberstrue
 

Info

Note: the internal ip address on line 50 must be the actual ip address of our host (not 127.0.0.1 as from the docker image, 127.0.0.1 is the docker image and not the host machine/server).

Add any other endpoints in the targets section of the yml.

Installing and configuring Grafana

Grafana is a great platform for visualising data and metrics from large data sets.  It can connect with a very large number of data sources and has native (built-in) prometheus support, which makes it extremely easy to integrate prometheus and provides an attractive and versatile front-end to view various prometheus metrics.

So let's go ahead and install grafana and do some basic setup.

Like previous approaches, we'll use a simple docker image and setup data-persistency via a docker-volume, set a port for grafana to use (in this example we use a custom port), as well as setup public (anonymous) viewing access.

We only need two files for all this - which you can get by cloning

https://gitlab.jaytaala.com/docker/grafana.git

Code Block
languagebash
git clone https://gitlab.jaytaala.com/docker/grafana.git

The repo contains:

  • run-container.sh : shell script that will run our docker command to install / configure grafana.
  • grafana.ini : single configuration file for grafana;
  • dashboard-blackbox.json : blackbox exporter dashboard I've customised see Adding monitoring dashboards;
  • dashboard-node_exporter.json : node_exporter dashboard I've customised see Adding monitoring dashboards;

See the Explanation of run-container.sh section for an explanation and several notes for these files.

Once you've cloned the repo, simply make the script executable (if it isn't already) and run the script with:

Code Block
languagebash
chmod +x run-container.sh
./run-container.sh

Docker will then download the latest official Prometheus Docker image, create and configure the container and then run said container. 

If things worked as they should you should be able to visit the basic Prometheus web interface at http:\\<INTERNAL-SERVER-IP>:10090.

Info

Replace <INTERNAL-SERVER-IP> with the actual internal IP address of your server, e.g. 192.168.1.x, 10.0.0.x (or whatever it is).

That's it!  You will now have Prometheus running on port 10090.

Info

Note: port 10090 is not the default Prometheus port (which is 9090) - I have another service using that port (which I didn't want to change).  See the Explanation of run-container.sh section for notes on changing the port.

...

This script simply creates (and/or runs) our grafana container.  So what's going on here?

View Git file
pathrun-container.sh
repository-id9
languagebash
branchrefs/tags/documented
linenumberstrue

...

removes / deletes the container.  This deletes the already running container (if it exists) before creating a new one.

Tip

Don't worry(!) our data won't be lost since we're persisting the grafana data with a Docker data volume!

...

main docker run command.

The -d flag means "detached" (i.e. run it detached or in the background) and the --name=graf names the container that will be created "graf".

...

this data-binds our grafana.ini config file to /etc/grafana/grafana.ini in the created Docker container.

...

this instructs Docker to create a data volume (for storing our persistent grafana data) and call it "graf-data".

It's kind of similar to a data-bind but it's managed by Docker and provides many advantages.

The actual files (and persistent data) can be found at:
/var/lib/docker/volumes/graf-data/_data

...

binds port 4000 on the host to post 4000 on the container.

Info

Note: this is not the default port for grafana (default port is 3000).  I had to change it as I was running another service on port 3000.  Change both of these to whatever port you would like to bind grafana to.

You'll also need to modify the included grafana.ini file and set http_port to whatever port you want to use.

...

This file is the main grafana config file for our setup.  It defines things server related settings, as well as settings we'll use to enable public viewing access.

View Git file
pathgrafana.ini
repository-id9
branchrefs/tags/documented
linenumberstrue

...

Settings to enable anonymous (public) viewer access to any dashboards you give Viewer access to.

...

A bit about Blackbox Exporter, Node_Exporter and Grafana...

Expand

Blackbox Exporter

Blackbox exporter is a prometheus exporter which can probe endpoints via http/https, icmp, etc.  It's extremely simple to setup and can be used to monitor the status of various endpoints (which could be web applications, REST endpoints, etc.).

We've already setup blackbox-exporter in our prometheus configuration file (see lines 36 to 57). 

Node_Exporter

node_exporter is a prometheus exporter which monitors hardware and 'nix OS metrics.  It runs on a port exposed to prometheus and prometheus can then query it and get a (large) ranger of metrics for whatever machine is running node_exporter.

Grafana

Grafana is a great platform for visualising data and metrics from large data sets.  It can connect with a very large number of data sources and has native (built-in) prometheus support, which makes it extremely easy to integrate prometheus and provides an attractive and versatile front-end to view various prometheus metrics.

Guide

We'll cover the following steps:

Table of Contents
minLevel3

Installing and configuring Prometheus, Blackbox exporter, and Grafana with Docker-Compose

I'm assuming you are running an up-to-date Linux distro and you have Docker and Docker Compose installed (if not, you'll need to install Docker and then Docker Compose).

Thanks to Docker and Docker Compose, we'll only need to run a single command to deploy Prometheus, an exporter (which can query endpoints), and Grafana.  Let's start by cloning our setup files from my git repo:

https://gitlab.jaytaala.com/docker/docker-compose-prometheus-grafana

Code Block
languagebash
git clone https://gitlab.jaytaala.com/docker/docker-compose-prometheus-grafana.git

The repo contains several files:

  • blackbox-exporter.yml : configuration file for blackbox-exporter
  • dashboard-blackbox.json : blackbox exporter dashboard I've customised see Adding monitoring dashboards;
  • dashboard-node_exporter.json : node_exporter dashboard I've customised see Adding monitoring dashboards;
  • docker-compose.yml : docker compose config file
  • grafana.ini : grafana configuration file
  • prometheus.yml : prometheus configuration file

Once you've cloned the repo, simply run (from the with the cloned folder:

Code Block
docker-compose up -d


If things worked as they should you should be able to visit the basic Grafana web interface at http:\\<INTERNAL-SERVER-IP>:3000.

Info

Replace <INTERNAL-SERVER-IP> with the actual internal IP address of your server, e.g. 192.168.1.x, 10.0.0.x (or whatever it is).

That's it!  that wasn't so bad was it?

Troubleshooting

Expand

If it fails it's most likely that port 3000 on the host is taken - in which case simply change the "ports" first value in docker-compose.yml to another port.  E.g. for me I used:

Code Block
...
    ports:
      - 4000:3000
...


Optional reading (explanation of configuration files)

Expand

Anchor
prometheus.yml
prometheus.yml
 Explanation of prometheus.yml

This file is the main Prometheus.yml scrape configuration file.  It defines things like how often Prometheus should scrape (query) data and from where (and how) it should scrape this data.

View Git file
pathprometheus.yml
repository-id10
branchrefs/tags/documented
linenumberstrue

Line(s) numberComment
1-5Default scraping config for how often Prometheus should scrape.  Note, these can be overridden for specific scrape configs (see from line 22).
23-29

Prometheus (be default) monitors itself and can report on scrape query duration, samples added to db, data compaction stats etc.

31-34

Scrape configutation for node_exporter, an exporter which scrape server machine metrics.  We haven't set this up yet but will in the following sections.

Info

Note that 10.0.0.137:10091 refers to the internal IP address of the machine you want to monitor (and the port on which node_exporter is running). 

This is NOT localhost (localhost in this context is the Prometheus Docker container).


36-51

Scrape configuration for blackbox, a prober that can probe endpoints over HTTP, HTTPS, DNS, TCP and ICMP.  We haven't set this up yet but will in the following sections.

Anchor
grafana.ini
grafana.ini
 Explanation of grafana.ini

This file is the main grafana config file for our setup.  It defines things server related settings, as well as settings we'll use to enable public viewing access.

View Git file
pathgrafana.ini
repository-id10
branchrefs/tags/documented
linenumberstrue

Line(s) numberComment
1-2Defines server settings for our setup - including the port to bind/ and our domain name.
4-9

Settings to enable anonymous (public) viewer access to any dashboards you give Viewer access to.


Info

You'll note that I've defined my domain as mon.jaytaala.com.  This is an external address for which I'm using a reverse proxy to secure (SSL) and route traffic to the internal 4000 port of Grafana.  See Apache reverse-proxy SSL to multiple server applications for more information on how to implement this.


Installing and configuring node_exporter (to monitor server stats)

By default node_exporter enables a large number of "collectors" (modules which collect certain information from the machine).  See here for a list of collectors enabled by default (and what info they collect).

node_exporter can be run from a docker container, but it's not recommended since it should be run directly on the host hardware to collect stats.

Installing node_exporter can be done by downloading a recent version version, untar'ing and executing.  We're going to be doing an extra step here to manage node_exporter with systemd (so it starts on server boot etc.).

We start with downloading.  You can find a link for the latest version at https://prometheus.io/download/#node_exporter.  At the time of this writing the latest stable version for linux-amd64 was node_exporter-0.18.1.linux-amd64.tar.gz.

We'll download, untar, and then move it to /opt/node_exporter

Code Block
languagebash
wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
tar -xf node_exporter-0.18.1.linux-amd64.tar.gz
sudo mv node_exporter-0.18.1.linux-amd64 /opt/node_exporter

With node_exporter installed let's make it easier to manage by creating a systemd service.  Create a file in /etc/systemd/system/ with your favourite text editor (I'm using vim here):

Code Block
languagebash
sudo vim /etc/systemd/system/node_exporter.service

and paste the following:

Code Block
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/opt/node_exporter --web.listen-address=:10091

[Install]
WantedBy=multi-user.target


Info

I'm using a non-default port here (10091) so change it to whatever port you prefer (or have free).

Finally let's enable it (to start on boot) and start the service

Code Block
languagebash
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

We've already setup node_exporter in our prometheus configuration file (see lines 31 to 34). 

View Git file
pathprometheus.yml
lastline34
repository-id10
firstline31
branchrefs/tags/documented
linenumberstrue
 

Setting up Grafana data sources and dashboards

Adding Prometheus as a data source to Grafana

Once you have your docker container containers running, access port 4000 3000 on the server with a browser (e.g. visit http://192.168.0.1:4000 3000 or whatever your internal address is for your server). 

...

Grafana will grab the dashboard and after a second or so will fetch the dashboard ready for importing.  Enter a few options (see arrows) and then click "Import":

References

  1. https://www.docker.com/
  2. https://docs.docker.com/compose/
  3. https://prometheus.io/
  4. https://github.com/prometheus/node_exporter
  5. https://prometheus.io/download/#node_exporter
  6. https://github.com/prometheus/blackbox_exporter
  7. https://grafana.com/

...