This is the third article in this series covering how to install Bosun in a production environment. The first two articles covered the installation of Bosun and then configuring TSDBRelay to relay and aggregate data for us. This article is going to explore how we can load balance between multiple TSDBRelay instances with HAProxy.
Unfortunately Bosun does not support being ran in a load balanced configuration which is not great when we need high availability; I have discussed this with Bosun’s creator Kyle Brandt and there is also a Pull Request proposing changes to support a Bosun cluster but this is yet to be merged in. So what do we do if we don’t want to worry about Bosun being there when we send data? As discussed in the previous article we can utilise TSDBRelay to get Bosun out of the ‘crtical path’ by forwarding the data directly to our OpenTSDB server however because our applications will send data to TSDBRelay instead of Bosun we need to make TSDBRelay highly available. Fortunately TSDBRelay does support being ran in a load balanced configuration so we are going to add a second (you can add as many as you wish) TSDBRelay instance and then use HAProxy to load balance between them.
Create second TSDBRelay instance
Let’s start by creating a second TSDBRelay instance! I’m not going to go into too much detail about how to do this as the process is covered in depth in the previous article. The basic process is create an Ubuntu instance (or other distro of your liking), install Go 1.11.2, pull TSDBRelay from GitHub and finally install and configure Supervisor to run TSDBRelay. Ideally I’d recommend having at least three instances but you are really free to add as many as you want.
Install and Confiure HAProxy
Create a Linux VM using your distro of choice (as usual I am using Ubuntu 18.04), SSH into it and run apt-get update to make sure everything is up-to-date.
sudo apt-get update
Once done let’s install HAProxy
sudo apt-get install haproxy
Once HAProxy is downloaded and installed we need to enable the init script, this will allow HAProxy to autostart. Run the following command to edit the script with Nano.
sudo nano /etc/default/haproxy
Append Enabled=1 to the file and then save and close (ctrl+o, enter, ctrl+x).
Now let’s open the config file using nano and start configuring our load balancer.
sudo nano /etc/haproxy/haproxy.cfg
Firstly let’s set the default mode to tcp and option to tcplog. Update these in the ‘defaults’ section.
mode tcp option tcplog
At the bottom of the file we are now going to add a frontend, this will instruct HAProxy what to listen and which backend to use. Append the following to the bottom of the config file.
frontend tsdbrelay bind 0.0.0.0:5252 default_backend tsdbrelay-backend
This config is just telling HAProxy to listen on port 5252 and use the backend ‘tsdbrelay-backend’ which we will now specify.
backend tsdbrelay-backend balance roundrobin mode tcp server tsdbrelay1 10.0.0.8:5252 check server tsdbrelay2 10.0.0.10:5252 check
Set the tsdbrelay1 and 2 ip addresses and ports to the ip address and ports of your TSDBRelay instances, if you are using more than two instance then keep adding servers to the file until they are all specified.
If your HAPRoxy instance is not externally facing then you could enable the stats dashboard by adding the following configuration to the configuration file, this is not really necessary but it’s nice to see what HAProxy is doing. Make sure you replace admin:pass with strong credentials.
listen stats bind :80 mode http log global maxconn 10 clitimeout 100s srvtimeout 100s contimeout 100s timeout queue 100s stats enable stats hide-version stats refresh 30s stats show-node stats auth admin:pass stats uri /haproxy?stats
A complete copy of my HAPRoxy configuration file can be found here.
Once you are happy with your configuration save and exit and then restart HAProxy
sudo service haproxy restart
Now instead of sending data to TSDBRelay we can send data to HAProxy which will then load balance the data between our TSDBRelay instances.