What do you think? Discuss, post comments, or ask questions at the end of this article [More about me]

You might have need to create a secure, persistent connection between two servers over an (unsecured) network (like the internet).

An actual example of this was when I needed to scrape prometheus metrics to my home server from an VPS server I have with a cloud provider (which is many miles kms away from where I am).  Now I could have just opened a port to the prometheus exporter on the VPS server... but that would be insecure (i.e. scrape metrics with be traversing over the internet unencrypted).

So basically I needed to connect these two separated servers securely as if they were on the same network.  Connecting these via a VPN is a good choice here, but another (simpler?) option is to create an SSH tunnel between the servers - especially if you already SSH into both servers.

Below we'll create a simple systemd service that will create an SSH tunnel between two servers and restart itself if the connection is severed.

Guide

Pre-requisites

For this approach you'll need:

  • to be able to SSH (by key, not password) into one of your servers (server B) from the other server (server A). 
  • a disto running systemd.

Create a systemd service to take care of everything

Let's create a systemd service file (call it what you want) in /etc/systemd/system:

sudo vim /etc/systemd/system/ssh-tunnel-persistent.service

I'm using  vim  here but you can use whatever editor you like (e.g. nano, if you must...).

and copy/paste (and modify) the following:

[Unit]
Description=Persistent SSH Tunnel to from port 9092 on this server to port 9090 on external server (for encrypted traffic)
After=network.target

[Service]
Restart=on-failure
RestartSec=5
ExecStart=/usr/bin/ssh -NTC -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes -L 0.0.0.0:9092:127.0.0.1:9090 user@93.184.216.34

[Install]
WantedBy=multi-user.target

You can lookup the options for ssh that I'm using here. 

You'll also note that I'm binding on all interfaces (0.0.0.0) on server A - this may not be necessary for your setup but I was running prometheus from a docker container and access the host server (server A) via it's internal network address.

Change user@93.184.216.34 to the user and @server address you use to ssh into server B.

Now let's enable and start our service:

sudo systemctl daemon-reload
sudo systemctl enable ssh-tunnel-persistent.service
sudo systemctl start ssh-tunnel-persistent.service

That's it.  It things went well you can send traffic to localhost port 9092 and securely SSH tunnel that traffic to port 9090 on server B.  Furthermore, if the connection is dropped, then systemd will wait 5 seconds and continuously attempt to reconnect.

References

  1. https://linux.die.net/man/5/ssh_config
  2. https://nitori.org/creating-a-persistent-ssh-tunnel-as-a-systemd-user-service.html