February 27, 2011
Setup Pound, Varnish & Apache w/ Multiple IPs & BackEnds
The following is a tutorial in setting up Pound (SSL Wrapper) that is being proxied to Varnish (Cache) which is finally proxied to Apache. This setup also allows for multiple IP Addresses (for SSL Cert) and multiple Varnish Backends.
This article assumes you have working knowledge of Linux and CentOS distribution in particular. This installation is based upon the “yum” paths for all instances. It also assumes you have apache setup already, serving content with multiple IP Addresses and your server’s external IP Address 1 = 2.3.4.5 & IP Address 2 = 6.7.8.9
———————————————————————————————————
Pound SSL Wrapper
Pound is a reverse proxy, load balancer and SSL wrapper that allows you to listen on a secure port (443), decrypts the SSL Certificate and then sends the decrypted data to a local non-ssl port (80) to be served by the Varnish. First, let’s setup the pound configuration.
yum install pound
edit /etc/pound.cfg
ListenHTTP
Address 2.3.4.5
Port 80
Service
Redirect "https://www.mydomain.com"
End
End
ListenHTTPS
Address 2.3.4.5
Port 443
Cert "/var/certs/mydomain.com.pem"
Service
BackEnd
Address 127.0.0.3
Port 80
End
End
End
In this example, ListenHTTP, we want to make sure that the entire site is served securely and redirecting 80 traffic to 443. In ListenHTTPS, we pass it the path to the cert in .pem format. Not only is this required for ListenHTTPS, but it MUST be in .pem format.
To convert a crt to a pem, run the following command (replacing files with your own)
“cat mydomain.com.key mydomain.com.crt (mydomain.com.ca) > mydomain.com.pem”
Varnish Caching
Varnish is a Reverse Proxy (Caching, HTTP Accelerator) that allows for Load Balancing.
yum install varnish
edit /etc/sysconfig/varnish
I used Alternative 2 for the configuration. Set the IP(s) in the -a option. While the IPs are comma separated, make sure there isn’t a space between the IP addreses. It will fail. Pound is listening for the IP Address 2.3.4.5, so we don’t need to have Varnish listen for that IP. We need to have it listen on the port that pound specified as its backend. In this case, 127.0.0.3:80. Since pound is not serving secure content for IP Addresss 6.7.8.9, we can have varnish listen for that IP.
## Alternative 2, Configuration with VCL
DAEMON_OPTS="-a \"6.7.8.9:80,127.0.0.3:80\" \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-u varnish -g varnish \
-s file,/var/lib/varnish/varnish_storage.bin,1G"
Next, we need to edit the default vcl file for Varnish which is where all the settings will be added.
/etc/varnish/default.vcl
# For IP Address 6.7.8.9 and the rest of the domains on the server.
backend default {
.host = "127.0.0.2";
.port = "80";
}
# For IP Address 2.3.4.5 listening on Pound, proxied to varnish on 127.0.0.3. Setting Apache Backend to 127.0.0.4
backend mydomain{
.host = "127.0.0.4";
.port = "80";
}
sub vcl_fetch {
# Add a unique header containing the cache servers IP address:
remove obj.http.X-Varnish-IP;
set obj.http.X-Varnish-IP = server.ip;
}
sub vcl_recv {
# choose a backend depending on domain
if (req.http.host ~ "www.mydomain.com") {
set req.backend = mydomain;
}else{
set req.backend = default;
}
# remove and reset the clients IP address
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For=client.ip;
}
In the sub vcl_recv and sub vcl_fetch, I am doing two things. 1. Setting the backend server depending on which domain it is looking for. 2. Passing along the client’s IP Addresss to reflect in the apache log files.
SSL Conf
edit /etc/httpd/conf.d/ssl.conf
Pound is now listening on port 443, which means you can’t bind the ssl.conf to 443 as well. We need to comment out the listener.
# Listen 443
httpd.conf
/etc/httpd/conf/httpd.conf
We need to set apache to listen and set the NameVirtualHost on the ports that we specified in /etc/varnish/default.vcl
Listen 127.0.0.2:80 Listen 127.0.0.4:80 NameVirtualHost 127.0.0.2:80 NameVirtualHost 127.0.0.4:80
Vhosts File
/etc/httpd/sites-available/mydomain.com.conf
Finally we need to change the virtualhost IP Address in the vhost.conf to listen to the same ports as above depending on domain.
<virtualhost 127.0.0.2:80> ... </virtualhost> or <virtualhost 127.0.0.4:80> ... </virtualhost>
If you need the client’s IP Address (I don’t know why you wouldn’t), you need to add the following into the virtualhost above. Replace with your log file paths
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" varnishcombined
CustomLog /www/mydomain.com/logs/access.log varnishcombined
Now that we have finished the setup, this will require a restart of all services for it to work. If it was setup properly, they should all say “ok”.
/etc/init.d/pound start (restart)
/etc/init.d/varnish start (restart)
/etc/init.d/httpd restart
Monitoring Varnish Stats, Logs and Histogram
Based on install paths, the following should work. From the prompt: varnishstat, varnishlog, varnishhist
Varnishstat and Varnishlog give detailed information on headers being passed and the status of the cache. Varnishhist is a histogram of the traffic coming to your site. The left side is cached content and the right side is server generated content.
Stress Testing New Setup
If you have very little traffic, you might need to use a stress test tool to find out how the server will handle your new setup. Siege is a tool that forces many concurrent connections. If you run this tool before implementing Varnish, it may but more likely will bring down your server. The load average will shoot up very quickly (to 80.00+) until the server can’t handle it and stops responding. After installing Varnish, you can run up to thousands of concurrent connections and the load average won’t go past 10.00 (depending on many factors).
The parameters being passed to siege below, -c350 (350 is the number of concurrent connections, default is 15), -t1M (1M is the total time that siege will run for, 1 minute)
From command prompt, type:
siege -c350 -t1M www.mydomain.com
* It helps to have multiple shells up and watch “top”, “varnishstat” and “varnishhist”.
Log Files
tail -f /var/log/messages and /var/log/varnish/varnish.log
