Intelligent load balancing with haproxy

At one company I work for, we have 3 servers doing only frontend work (serving HTTP requests) and 4 other servers doing only background jobs. Those are very powerful servers with many cores and tens of gigabytes of RAM. The problem with dedicated servers is that background job servers are underutilized most of the time and although our frontends could handle 2-3x the traffic, there’s never too much capacity for handling bugs in frontend code (causing a lot of requests coming in) or DoS attacks.

Problem: we can’t do rate limiting. Our application is special in a way, because our frontend requests are very computational heavy and we don’t do much caching (which would make us do a lot of precalculations, causing us to invest multiple times into hardware). Due to the majority (but not all) of requests being very slow (300-400 ms) and most clients coming from the same IP address, we can’t really do rate limiting. Or at least, do it in a simple way. Say, we rate limit requests per user, but then either the limit is too small (and the fast requests like logging in or API calls (at some times we can have more fast requests than slow ones) become slow, as they are delayed) or too large (and the slow requests can easily kill the whole cluster, as we don’t have the capacity to handle more than 2-3x of normal load).

Rate limiting is crucial to any application having many clients. Other than hacker induced DoS attacks, sometimes a JavaScript anomaly happens on a friendly customers’ computer and their browser starts sending the same slow request in quantities of thousands per second. Since we don’t have the capacity to handle such amount of requests, nor rate limiting per client, our application goes down for all clients.

In other words, we don’t want to deploy extra hardware just to have the capacity to handle DoS attacks and nothing more (user experience does not change directly because of that, in our case). But we want to be able to mitigate such attacks automatically. The striking thing is that we already have unused hardware for most of the day – background job servers are not under high load most of the time.

The solution with haproxy
This is where haproxy comes in. We decided to add the extra capacity by giving out the frontend tasks to all servers instead of just the frontend ones. We changed nothing from background job implementation side, we simply launched frontends on all servers and started serving requests from all of them. But that wouldn’t be ideal without haproxy’s agent-check ability.

On each frontend server, there is a simple systemd maintained haproxy agent:

#!/usr/bin/env python3
import socket
import psutil
import re

TCP_IP = ''
TCP_PORT = 5001

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((TCP_IP, TCP_PORT))

def get_load():
  sidekiq_count = 0
  busy_count = 0
  for proc in psutil.process_iter():
      pinfo = proc.as_dict(attrs=['pid', 'name', 'cmdline'])
      if ((pinfo['name'] == 'ruby') and ('^sidekiq', pinfo['cmdline'][0]))):
        sidekiq_count += 1
        if not'\[0 of ', pinfo['cmdline'][0]):
          busy_count += 1
    except psutil.NoSuchProcess:
  if busy_count >= 24:
    load = 0
    load = round(100*(1-busy_count/24))
  return load

while 1:
  conn, addr = s.accept()
  print('Connection address:', addr)
  data = str.encode(str(get_load()) + '%\r\n')

This script is just an example of what you could do with agent-checks in haproxy. I’ve seen some people doing haproxy management panels through agent-checks, but it was even better to automate everything, in our case. This program responds to TCP connections with a percentage of how much an haproxy backend weight should be modified (which we set to 100 as a baseline for all backend servers (instead of default 1), so that the end result would be a percentage). When calculating weight (the percentage), it takes the amount of Sidekiq (background job daemon) process count and busy Sidekiq process count. Then, if the busy count of Sidekiqs is more than the number of physical cores in the system (e.g. 24, hardcoded), it removes the system from the pool by setting weight to 0%. If there are less than 24 busy Sidekiqs, it reduces the weight proportionally.

haproxy is configured to round robin requests, assuming that they are all more or less equally the same (and we also assume that IO is not a problem for our application, which is most of the time true), thus, load is always distributed more or less equally across the pool. We never send requests to servers doing lots of background job processing at the time and since some of the servers don’t have as many Sidekiqs as CPU cores, we always have some frontend processing power, regardless of how many Sidekiqs are actually busy. Last, but the best – Sidekiq’s scheduling a bit vague to me and I can see that it sometimes schedules x jobs to worker A, z jobs to worker B and etc. One irrelevant weird thing is that x or z is sometimes >1, even though there are other available workers. The other issue, very relevant to me, is that sometimes Sidekiq can schedule some jobs for the worker on frontend servers and background job servers would be under less load than frontends – thus, being a better choice for serving frontend requests at the time. Such agent-checks automate everything, giving us a lot more capacity to handle client-side anomalies while using same hardware.

Transferring files from Linux to iPhone over USB

If you’re like me, being a Linux and an iPhone user is a bit of a pain. The only options to transfer files from your computer you have are all over a wireless network, limiting you to speeds you get either from your carrier or a router the least.

I don’t have a good router. Even though it is supposed to give me 300 Mb per second speed, being 1 meter away from the router gives only around 50 Mb per second which is roughly 6 MB/s.

If you’re transferring files from a device connected over WiFi to an iPhone which is also connected over WiFi, I’m not getting even those 6 MB/s.

So, in a situation like this, transferring files over USB is a life saver. This simple trick/hack will help you get better speeds or security by not using WiFi for file transfers.

This article focuses on transferring media to VLC, but only because this is what I need mostly. It should work for any other app, which has file sharing server functionality (uploading files over a browser), like Kingsoft Office Free.

The downside of this method is that while your computer is connected over personal hotspot, it can use out your mobile data for other purposes, like downloading updates. To minimise this, you should not use your computer for anything that might need internet traffic. Again – it’s NOT going to use your mobile data for transferring files.

First, you will need to get Personal Hotspot functionality working on your iPhone. Here’s a great how to:

Next, make sure that you get a working connection on your laptop using USB. Disconnect WiFi on your PC or unplug ethernet, if you’re using it, just to be 100% sure. This will ensure you have no IP routing problems, which may arise, because both your iPhone and your PC are essentially connected to the same network.

Optionally, turn off WiFi on your iPhone. This is just to drop the confusion whether you’re using WiFi to transfer files. And remember that even if your iPhone is connected to a WiFi network, Personal Hotspot will still be using mobile data for internet on your computer. But since we’re using local IP addresses, file transfers are happening over USB/local network, not internet.

Lastly, simply start transferring files over WiFi. Because iPhone owns the IP address you’re trying to reach from your computer, network traffic for file transfers will never leave your iPhone and you will get reasonable USB 2.0 speeds (which, in my case, were a good improvement over transferring over WiFi).

In my tests, I was using a non-Apple certified 3 meter USB cable, which might have had an effect on transfer speeds (I got only 10 MB/s, vs. 2 MB/s over WiFi (laptop to iPhone both on WiFI)), but it still showed a big improvement over my WiFi. Anyways, the point of this article is not improvement of transfer speeds per se (which you will see only if you are on a shitty WiFi like me), but the point that you could use this trick to transfer files from your computer to your iPhone even when WiFi is unavailable or you don’t want your files to be going through WiFi.

Even more…
For advanced users, you can connect your computer to WiFi and over USB at the same time, but make sure to check “Use this connection for resources only on its network” (in Network Manager network settings for iPhone, IPv4 Settings tab, “Routes…” button) and disconnect your iPhone from WiFi to get VLC listening on the Personal Hotspot IP address. This way, your laptop will be usable, because you won’t worry about spending those precious mobile data gigabytes on browsing on your laptop while transferring files.

SSL on GlusterFS

We run GlusterFS in our local environment for many things… Starting with file sharing server, screenshot directory for our CI and ending with Puppet folders, which help to make it highly available (although, I’m considering testing out master/slave configuration to see if GlusterFS performance is of any bottleneck for us).

Some history

Feel free to skip ahead if you just came here for a tutorial how to setup SSL for GlusterFS. The problem I ran into with GlusterFS was security and documentation. Things are vaguely documented and that made it difficult. Previously (about a year ago), there was almost 0 documentation about SSL support in GlusterFS. So I dug a bit deeper into it and got VLANs and IP address access controls working for it. But that doesn’t cut it, because I’m a cheapo and I run GlusterFS over our local gigabit network switches, which are shared amongst other servers. So, encryption was a necessity. I could have gone with OpenVPN to encrypt things, but I would then need it to be highly available and I would need a couple of additional containers/virtual machines for it to run. That’s a pain in the ass to maintain, if you keep it separated from other services.

How-to server

Back to SSL, first resource that you may find after some googling is a blog post by OneFellow. There are also 2 places in documention where the same stuff is repeated: [1] [2]. It’s generally a good place to start, but probably a bit outdated. I used GlusterFS 3.6.2 (from the official PPA repositories, on Ubuntu 14.04) and it was still in development at the time. Note that there is a bug in this version which prevents mounting volumes using volume files, but I fixed it (bug-1208676) and I hope it will be included in the next release. You can patch your system by modifying /sbin/mount.glusterfs accordingly.

To make everything convenient for you, I’ll repeat some things that OneFellow has noted already:

Start by creating an SSL key and certificate by running:

openssl genrsa -out glusterfs.key 1024
openssl req -new -x509 -key glusterfs.key -subj /CN=Anyone -out glusterfs.pem

Note that the CN (Common Name, which is “Anyone” in this example), which can later be used for authorization (works similarly to authorizing using IPs).

Some letters in GlusterFS mailing list mention that you can change SSL paths, others suggest that you cannot and they are hardcoded into your GlusterFS binaries.

Don’t forget to create file, which seems to be just a concatinated list of approved certificates, not an actual certificate authority certificate (because, obviously the certificate is self-signed). Look into ssl.cert-depth volume option, if you need something else. Anyways, you can create the CA file by copying glusterfs.pem to

cp glusterfs.pem

Next, distribute these files to all of the GlusterFS nodes and clients (you may or may not use separate certificates for different clients, but a single GlusterFS can use only one set of SSL certificates, because of the hardcoded paths). Don’t forget to chown it to root:root and chmod it to 600 for security purposes:

chown root:root /etc/ssl/glusterfs.*
chmod 600 /etc/ssl/glusterfs.*

Turn on SSL for volumes that you want to use encryption (volume name “gv0” in this example):

gluster volume set gv0 client.ssl on
gluster volume set gv0 server.ssl on

Set cipher list (for me, the default settings didn’t work, probably because of POODLE mitigation; I also need to note that I was running openssl 1.0.1f-1ubuntu2.11):

gluster volume set ssl.cipher-list ‘HIGH:!SSLv2’

Also, it could be that you need to set allowed CNs in auth.ssl-allow volume option (for me, I couldn’t mount volumes without this option and I reported bug-1209432, it seems we are missing a patch in 3.6 branch, so my guess is that this will be fixed in later 3.6 versions):

gluster volume set auth.ssl-allow ‘Anyone’

Next, stop and start the volume (you can do this on one host, if others are reachable, this is unnecessary to repeat on them):

gluster volume stop gv0
gluster volume start gv0

Lastly, restart glusterfs-server service on all hosts or just reboot the machines:

service glusterfs-server restart

If you’re facing difficulties, stop the volume, stop glusterfs-server on all servers and start everything up again. Check the logs on all nodes and clients. Also, check “gluster volume info” and “gluster volume status” to see if everything is up and okay. I was running into very weird problems (e.g. DNS issues, GlusterFS falling back to trying to resolve IPv6 addresses instead of IPv4) when I didn’t do everything in order, e.g. left something not restarted.

Don’t forget to give GlusterFS enough RAM (swap is not enough). In my case, I was experiencing weird behaviour with 512 MB assigned to GlusterFS nodes. Assigning 2G of RAM fixed everything, not to mention that GlusterFS started working a bit faster.

How-to client

Make sure that you have the same GlusterFS version on the client side (in my case, it’s 3.6.2, just to remind you). You can mount by using a volume file or using a single server. Don’t take me wrong – the single host will be used to retrieve all nodes and all nodes will be used when mounting the volume, not just that single one. As for volume files, in 3.6.2, you have to patch /sbin/mount.glusterfs yourself first. Then simply include the option “option transport.socket.ssl-enabled on”  in “volume xxx” paragraphs and remount (try unmounting first, then mounting):

volume files1
type protocol/client
option transport.socket.ssl-enabled on
option transport-type tcp
option remote-host
option remote-subvolume /data/puppet

Feel free to see my full volume file for reference here.


Hope this helps. If anything, feel free to comment.

Tweaks to default PuTTY settings

Recently, I stumbled upon which describes some useful PuTTY tweaks. However, I personally find that some additional settings may be needed for the perfect experience:

  • Enable logging at Session -> Logging. Logging is extremely useful. Now you automatically have a backup of all configs, logs and so on at no cost of doing anything extra. It’s very useful for extra forensics, too. Set Session logging to Printable output (so you wouldn’t record any passwords or so) and Log file name to something like &Y-&M-&D-&T-&H.log. This stands for full date and time of start of the session and a hostname. You don’t want to log to a single file, as it can get very big and sometimes you want to clean up only some stuff. Lastly, pick Always append to the end of it, if you want to log every session as you defined in Log file name.
  • Disable that annoying bell at Terminal -> Bell. Pick None (bell disabled) to make sure that it’s not beeping when, say, you overhit Backspace. This could be extremely annoying when working at night.
  • Increase lines of scrollback for a larger buffer at Window. The bigger, the better, the less you will need that log file for still open sessions.
  • Enable scrollbar in full screen mode at Window. This is extremely useful is you like to switch to fullscreen often and have a large scrollback buffer. Using a mouse to scroll through thousands of lines can be tricky without a scrollbar.
  • Increase readability with antialiasing or ClearType (taste preference) at Window -> Appearance. Just choose whichever you prefer at Font quality.
  • Enable shortcut to switch to fullscreen faster at Window -> Behaviour. Just ensure that Full screen on Alt-Enter is checked. Then once a session is active, you can easily isolate yourself from any distractions by hitting Alt+Enter and making PuTTY fullscreen.
  • Make PuTTY always on top of other windows, if you find yourself switching between windows often at Window -> Behaviour. All you need to do it to check Ensure window is always on top. This could be both annoying and useful, depending on your use. You can also change this setting for already active sessions by right clicking on PuTTY window title and choosing Change settings…
  • Safer pasting by changing mouse buttons bindings at Window -> Selection. Ever accidentally pasted something on PuTTY by right clicking? You can disable that! Just choose one of the Action of mouse buttons. Both non-default choices disable right click to paste action, Windows choice may be more convenient for Windows users due to right click functionality bringing up menu and middle click extending (but not pasting), whereas xterm choice will flatter Linux users with right mouse click extending and middle click pasting.

There’s plenty more, but if you do need it, I would strongly encourage reviewing all options PuTTY has to offer yourself. Hope this helps!

Why Docker is not amazing and why I still want to use it

First off, to start with, I am very young in this niche. I became interested in Docker some-when in May or April, 2014. Back then, it was still pretty young. Even as of writing this, Docker is only 2 years old. The sad part is that it had way more bug reports than jails had in the past 10 years or so and I was feeling it when deploying it over the past 2 months. But I still want to use it. In some cases more than others. Continue reading “Why Docker is not amazing and why I still want to use it”