HAProxy and Chef - dynamic backend server list

Blog calendar

RSS feed from Michal Frackowiak's blog

subscribe to the RSS feed

— or —

get my blog posts via email

michal-frackowiakmichal-frackowiak
SquarkSquark
shark797039shark797039
Arotaritei VladArotaritei Vlad
clearekicleareki
RefutnikRefutnik
TRT- Vipul SharmaTRT- Vipul Sharma
Matt GentileMatt Gentile
HirelawyerHirelawyer
Helmut_pdorfHelmut_pdorf
Sven StettnerSven Stettner
michalf23michalf23
GabrysGabrys
leigerleiger
srivercxsrivercx
Joshua DarbyJoshua Darby
lil g easylil g easy
Mr ShaggyMr Shaggy
Chen XXChen XX
Super Dr GreenSuper Dr Green

... and more

Watch: site | category | page

Blog tags

« Back to the Blog

20 Nov 2014 10:02

After the recent load balancer upgrade at Wikidot people asked me about the magic behind automatic HAProxy configuration — namely, how do we solve dynamical addition and removal of backend servers. It's not that complex, we use Chef.

Below I will tell you how we do it.

Assumptions:

  1. We have a working Chef server.
  2. All nodes run Ubuntu, but it's not really important.
  3. All nodes run chef-client periodically (in our case, every 3 minutes).
  4. All web nodes have role web.

As a result will develop a minimal haproxy cookbook to be run on the HAProxy node that:

  1. Will set up HAproxy service.
  2. Will discover and connect to all backend servers.

I will skip the parts related to setting up Chef, nuances in HAProxy configuration, multiple roles and cookbooks and concentrate just on the auto-configuration part. If this in any way encourages you to set up a similar environment, it's good :-)

Here are all important files in our haproxy cookbook:

haproxy
├── metadata.rb
├── recipes
│   └── default.rb
└── templates
    └── default
        └── haproxy.cfg.erb

default.rb

# Install HAProxy repo and package itself

apt_repository "haproxy_repo" do
  uri "http://ppa.launchpad.net/vbernat/haproxy-1.5/ubuntu"
  components ['main']
  distribution node['lsb']['codename']
  keyserver "keyserver.ubuntu.com"
  key "1C61B9CD"
  deb_src false
end

package 'haproxy'
package 'socat'

template "/etc/haproxy/haproxy.cfg" do
  source "haproxy.cfg.erb"
  owner "haproxy"
  group "haproxy"
  variables({
    backend_nodes: search(:node, "chef_environment:#{node.chef_environment} AND role:web").sort_by{ |n| n.name }
  })
  notifies :reload, 'service[haproxy]'
end

service "haproxy" do
  supports status: true, restart: true, reload: true
  action [ :enable, :start ]
end

What it does is that it adds the Vincent Bernat's HAProxy repo first. You might skip it if you are fine with HAProxy in your distribution. We want 1.5 badly.

Then it installs haproxy and socat (useful for admin stuff) packages.

The next lines create the config file from an ERB template. The important line is:

backend_nodes: search(:node, "chef_environment:#{node.chef_environment} AND role:web").sort_by{ |n| n.name }

Chef-client, when running this cookbook, contacts the Chef-server and does a search over instances asking for all nodes running in the same chef_environment and which role include web. We sort the results to make sure the order is consistent between runs. We pass this data to the template.

We can verify that the query actually returns the nodes we expect:

$ knife search node 'chef_environment:production_2 AND role:web' | grep -E '(Name|Roles)'
Node Name:   i-d15eda3b
Roles:       web
Node Name:   i-6ec4088f
Roles:       web
Node Name:   i-6dc5098c
Roles:       web
Node Name:   i-3491e1de
Roles:       web
Node Name:   i-2d0e81cc
Roles:       web
Node Name:   i-ce5fdb24
Roles:       web
Node Name:   i-d4c40835
Roles:       web
Node Name:   i-cd5fdb27
Roles:       web

One more thing — whenever the config file changes, we need to reload haproxy service. It's done by the line:

notifies :reload, 'service[haproxy]'

This way, whenever list of backend servers change, or we provide a new version of template which affects the config file, haproxy gracefully reloads it's config.

The service… part defines the haproxy service and makes it run by default.

haproxy.cfg.erb

global
  # Global config goes here

defaults
  # Defaults go here

frontend http-in
  bind *:80
  default_backend http-backend

backend http-backend
  balance roundrobin
  http-check expect status 200
  option httpchk GET /ping.php

  <% @backend_nodes.each do |node| %>
  server <%= node.name %> <%= node.ipaddress %>:80 check fall 1
  <% end %>

I have simplified the config a bit, but the essentials part are here. In the real config we have stuff like SSL termination, stats, several backends, throttling rules for abuse etc.

What's critical is that the list of backend nodes are created from the @backend_nodes variable. And you know, that's it! The only thing left is add the cookbook to the haproxy node and drive traffic to it.

haproxy.jpg

Now it's important to run chef-client periodically on your HAProxy node. This way your list of backend servers would stay up-to-date. One thing I have not mentioned — it helps if you periodically remove dead nodes from Chef server.

We have been running this setup for 2 weeks now on AWS and it works really, really well. We have 2 HAProxy nodes listed in DNS for *.wikidot.com with Route53 health check.

If you have any questions, feel free to ask!


rating: 0, tags: chef haproxy wikidot

rating: 0+x

del.icio.usdiggRedditYahooMyWebFurl

Add a New Comment
asdad