Blog calendar
— or —
... and more
Blog tags
20 Nov 2014 10:02
After the recent load balancer upgrade at Wikidot people asked me about the magic behind automatic HAProxy configuration — namely, how do we solve dynamical addition and removal of backend servers. It's not that complex, we use Chef.
Below I will tell you how we do it.
Assumptions:
- We have a working Chef server.
- All nodes run Ubuntu, but it's not really important.
- All nodes run chef-client periodically (in our case, every 3 minutes).
- All web nodes have role web.
As a result will develop a minimal haproxy cookbook to be run on the HAProxy node that:
- Will set up HAproxy service.
- Will discover and connect to all backend servers.
I will skip the parts related to setting up Chef, nuances in HAProxy configuration, multiple roles and cookbooks and concentrate just on the auto-configuration part. If this in any way encourages you to set up a similar environment, it's good :-)
Here are all important files in our haproxy cookbook:
haproxy
├── metadata.rb
├── recipes
│ └── default.rb
└── templates
└── default
└── haproxy.cfg.erb
default.rb
# Install HAProxy repo and package itself
apt_repository "haproxy_repo" do
uri "http://ppa.launchpad.net/vbernat/haproxy-1.5/ubuntu"
components ['main']
distribution node['lsb']['codename']
keyserver "keyserver.ubuntu.com"
key "1C61B9CD"
deb_src false
end
package 'haproxy'
package 'socat'
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
owner "haproxy"
group "haproxy"
variables({
backend_nodes: search(:node, "chef_environment:#{node.chef_environment} AND role:web").sort_by{ |n| n.name }
})
notifies :reload, 'service[haproxy]'
end
service "haproxy" do
supports status: true, restart: true, reload: true
action [ :enable, :start ]
end
What it does is that it adds the Vincent Bernat's HAProxy repo first. You might skip it if you are fine with HAProxy in your distribution. We want 1.5 badly.
Then it installs haproxy and socat (useful for admin stuff) packages.
The next lines create the config file from an ERB template. The important line is:
backend_nodes: search(:node, "chef_environment:#{node.chef_environment} AND role:web").sort_by{ |n| n.name }
Chef-client, when running this cookbook, contacts the Chef-server and does a search over instances asking for all nodes running in the same chef_environment and which role include web. We sort the results to make sure the order is consistent between runs. We pass this data to the template.
We can verify that the query actually returns the nodes we expect:
$ knife search node 'chef_environment:production_2 AND role:web' | grep -E '(Name|Roles)'
Node Name: i-d15eda3b
Roles: web
Node Name: i-6ec4088f
Roles: web
Node Name: i-6dc5098c
Roles: web
Node Name: i-3491e1de
Roles: web
Node Name: i-2d0e81cc
Roles: web
Node Name: i-ce5fdb24
Roles: web
Node Name: i-d4c40835
Roles: web
Node Name: i-cd5fdb27
Roles: web
One more thing — whenever the config file changes, we need to reload haproxy service. It's done by the line:
notifies :reload, 'service[haproxy]'
This way, whenever list of backend servers change, or we provide a new version of template which affects the config file, haproxy gracefully reloads it's config.
The service… part defines the haproxy service and makes it run by default.
haproxy.cfg.erb
global
# Global config goes here
defaults
# Defaults go here
frontend http-in
bind *:80
default_backend http-backend
backend http-backend
balance roundrobin
http-check expect status 200
option httpchk GET /ping.php
<% @backend_nodes.each do |node| %>
server <%= node.name %> <%= node.ipaddress %>:80 check fall 1
<% end %>
I have simplified the config a bit, but the essentials part are here. In the real config we have stuff like SSL termination, stats, several backends, throttling rules for abuse etc.
What's critical is that the list of backend nodes are created from the @backend_nodes variable. And you know, that's it! The only thing left is add the cookbook to the haproxy node and drive traffic to it.
Now it's important to run chef-client periodically on your HAProxy node. This way your list of backend servers would stay up-to-date. One thing I have not mentioned — it helps if you periodically remove dead nodes from Chef server.
We have been running this setup for 2 weeks now on AWS and it works really, really well. We have 2 HAProxy nodes listed in DNS for *.wikidot.com with Route53 health check.
If you have any questions, feel free to ask!
rating: 0, tags: chef haproxy wikidot
Is this the fix to the connectivity issues Wikidot had around a month ago?
Piotr Gabryjeluk
visit my blog
Piotr, that's right, it's a fix to this issue.
Apparently what we experienced at the time was related to a bug^H^H^Hissue with ELBs themselves, as I was told. The ELB team fixed it, but we decided to keep using HAProxy.
I have added several improvements like: deterministic backend selection for anonymous users (improves cache hits), weighting based on the actual zone in EC2 and a few other things. It's been working great so far :-)
Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me
BTW the link in the first line of the text doesn't work :-)
Piotr Gabryjeluk
visit my blog
"it helps if you periodically remove dead nodes from Chef server."
It''s better to avoid a Chef server all together in a AWS or cloud environment. It's just another component that can fail. You can use S3 to store Chef cookbooks. Use a simple script in user-data or cloud-init that pulls the latest cookbooks from S3. Use chef zero or solo to run the cookbooks. Run the same cloud-init script from cron.
I like your blog. I enjoyed reading your blog. It was amazing. Thanks a lot.
I like your blog. I enjoyed reading your blog. It was amazing. Thanks a lot.
I like your blog. I enjoyed reading your blog. It was amazing. Thanks a lot.
Thanks for sharing this nice article. and i wish to again on your new blog keep sharing the article.
Thanks For Share.
Thanks for sharing this nice article. I read it completely and get some interesting knowledge from this. I again thanks for sharing such a nice blog.
Thanks for sharing this nice article. I read it completely and get some interesting knowledge from this. I again thanks for sharing such a nice blog.