June 25, 2011
Posted at: 3:43 pm by Timothy Haroutunian
Categories: Tech and Web
Recently, we (my company) took over web hosting for one of our clients. On average, they get 500,000 visits per month and a high profile suite of sites. I setup a Cloud Server Farm for the client. One of their sites, is the biggest of the group and their flagship site. This site was built 2 years ago using Joomla, a Content Management System, by a third-party company. I am not an advocate of Joomla because of the trouble that we have had with it. It is a very resource heavy CMS and for me, the problems outweigh the benefits.
While setting up this site, everything was working great until I started to test the server under high stress using “Siege”. After running the test with 50 requests/second, the server started to buckle and after 15 seconds, Apache became unavailable. I spent about 2 weeks trying to figure out what could cause this problem, but until today, I have been unsuccessful. I will first explain what I tried and how each change I made, either did nothing or caused different problems making them not a viable solution.
My first thought was to optimize Apache to handle more connections at one time. Apache’s HTTP Server website has a Performance Tuning section. After making these changes and restarting Apache, the Stress Test still caused the server to die.
Next, I thought it might have been the mysql server that was the issue. Even though, the mysql server wasn’t getting high load, I decided to fine-tune mysql as well. Since we have a dedicated mysql server, I used the configuration that was located in mysql-huge.cnf. This configuration is designed for servers that are just used for mysql. These changes seemed to help, but it wasn’t even close to what the site should be performing at.
I then thought it might be the amount of RAM on the server. I resized the cloud server to 4G of memory, but there was still no change. I resized it back down to 1G of memory which is what we thought it should be at.
After looking into more server configurations as the problem and getting nowhere, I decided that I would see if Joomla was causing the issues. You should know that this is all being done on a development server. I started by removing the Javascript files that were being included to display the client’s Ads. They have a total of 32 JavaScript calls and takes 10 seconds to just load the Ads. Removing them helped speed up the site, but the high load server failures were still happening and I added their Ads back in.
Now, the server is using Varnish for caching the content. It should be holding onto a static version of the site for 2 minutes (what it defaults to), but it wasn’t keeping the cache. Apache was still getting called every time you load the site. Varnish was still pulling the page through cache, but it was not keeping the cache. I have Varnish running on all of our servers including my personal server farm, however, I couldn’t figure out why this site was not keeping cache.
I tried removing some of the third-party modules that could have been incorrectly programmed and causing serious performance issues. After removing all of the modules on the homepage, the problems still existed. I got to the point where I didn’t think that it was anything to do with the server or how the site was pulling its content through Joomla, but I felt that there was something on the Joomla Core that was the problem.
I spent the next few hours researching what Joomla could be doing that was making the server die. After finding this blog article, I had an idea. His problem wasn’t my problem, but something that he wrote got me thinking. What if, the problem is with the Joomla Cookies. I did a few more Google searches and found that Joomla does not handle cookies very well it uses cookies for everything Joomla does.
*Please read the rest of this before trying the following as this configuration did not work even though it was performing the way I thought it should.
This site is on a dedicated server and no other traffic will be going to it. It is a 64-bit Quad Core Centos Server with 1G of ram. In Varnish, I added the following to the VLC file.
sub vcl_fetch {
set beresp.ttl = 5m; # Time that the cache should be kept for.
unset beresp.http.set-cookie; # Unset all cookies on the site
}
I “sieged” the server again at 50 requests/second and sure enough, Varnish was keeping the cache and serving it out of Varnish and not hitting apache at all. I then proceeded to stress the server at 500 requests/second and still the server was holding strong. Finally, I stressed it at 800 requests/second and it was still functioning properly. This is more than what the client would receive for traffic on an average day and I was confident that this had solved the problem….read on!
I then tried to log into Joomla on the front-end and on the administrative side. As it turns out, removing the cookies was a serious problem for the functionality of Joomla and would not let me login. I then removed the change that I made to the VLC, restarted Varnish and tried to login again. This time it worked.
I thought about how I could work around this issue, but nothing was coming to me. I did more research and came across this blog article. It is a little bit hard to read, but once you make the configuration readable, I found this code. *Note that I changed the obj.http.set-cookie to beresp.http.set-cookie to account for the new version of the Varnish 3.0 language.
sub vcl_fetch {
set beresp.ttl = 5m;
if( req.request != "POST" ) {
unset beresp.http.set-cookie;
}
}
Once I added this code to the VLC and restarted Varnish, not only could I login to Joomla on the front-end and administrative side, but the server was still holding strong at 800 requests/second.
Now, I can only stress the server to the point where I think it will hold its own in a real life scenario, but that doesn’t mean other steps may need to be implemented if the site gets picked up by Digg or Google (Google is what took down this clients server last month). Until that happens, we can’t account for the amount of traffic the site will receive. One additional step you can take is to add a Load Balancer into the mix and allow for the additional traffic and relive some of the stress on the sole server. This however, will be a separate blog entry as this one is already long enough.
In closing, I hope this article helps you to save weeks of performance testing, server changes and headaches.
June 12, 2011
Posted at: 1:32 pm by Timothy Haroutunian
Categories: Rants and Randomness
Time Warner Cable has made me very unhappy because they make money on screwing over the customer. I have had problems with my service over the past two years and they will not admit that it is their fault, even though the technician who came out to look at my service told us there was a problem. It was a problem with their line and it has been that way for two years. When I called to talk to someone about it, they told me there was no record of the problem.
Due to no record of what they fixed, I can’t call the billing department to get a partial refund for the past two years of service. Every time it rains, we lose internet and TV service for the majority of channels. When they setup our service, they told us there wasn’t a good connection to our apartment. Instead of looking into the problem, they said it was wiring in our apartment that was the problem. It turns out that there was a slice of wire missing from a line down the street which was causing the connection problems every time it rained. I would switch to Satellite, but to get “local channels” like NESN, it would cost me an extra $40 a month to go to the next level up.
Does anyone else have problems with Time Warner Cable?
March 19, 2011
Posted at: 11:02 pm by Timothy Haroutunian
Categories: Uncategorized
*While writing this post, I realized that Varnish has been updated to 2.1.5 (not in EPEL yet) and this configuration will now break if setup.
If you have not yet setup Varnish or don’t understand the basics of Varnish, please read my post “Setup Pound, Varnish & Apache w/ Multiple IPs & BackEnds”. This article also assumes you understand the basics of a Firewall.
At work, I am setting up a new server environment for a client on our Cloud Servers from Rackspace. We have come up with our first approach to their needs, which is the following. One (1) Firewall Server using Varnish as a reverse proxy for web content to the other servers, Two (2) Production Web Servers running Varnish for Caching, One (1) Production Database Server, One (1) Development Web/Database Server.
Since I am still in the process of setting it up, I may not have it all figured out yet, but the basic concept is this. All DNS records for each site point to the IP Address of the Firewall Server. Next, set varnish to listen on that IP. You have two options for setting up the backend servers.
- Use rsync to synchronize the two production web servers. Then use Varnish as a load balancer to send equal or weighted amount of traffic to different servers.
- Use Varnish to proxy specific websites to different backend servers.
I am implementing option 2 into this environment. We have one website (high traffic) that will be on its own server and the client’s other sites will be on the second production server. On the backend server’s varnish configuration, you will change your listening IP to the IP address listed on the backend on the Firewall server. If you have a private IP address associated with each server, I would recommend using it as the IP addresses since those are faster and usually your hosting provider allows unlimited transfer between private IPs.
On the firewall server varnish vcl (Note: these aren’t the real private IP Addresses)
backend mydomain{
.host = "192.168.1.2";
.port = "80";
}
backend otherdomain{
.host = "192.168.1.3";
.port = "80";
}
sub vcl_recv {
if (req.http.host ~ "mydomain.com") {
set req.backend = mydomain;
}elseif (req.http.host ~ "otherdomain.com") {
set req.backend = otherdomain;
}
}
On the backend server’s varnish configuration (/etc/sysconfig/varnish)
DAEMON_OPTS="-a 192.168.1.2:80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-u varnish -g varnish \
-s file,/var/lib/varnish/varnish_storage.bin,1G"
DAEMON_OPTS="-a 192.168.1.3:80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-u varnish -g varnish \
-s file,/var/lib/varnish/varnish_storage.bin,1G"
The rest of the configuration would be your standard vcl from Varnish.
There are other solutions to what I am trying to do that might be much easier for the organization as a whole, but we are already using Varnish for caching on the server.
February 27, 2011
Posted at: 7:55 pm by Timothy Haroutunian
Categories: Uncategorized
The following is a tutorial in setting up Pound (SSL Wrapper) that is being proxied to Varnish (Cache) which is finally proxied to Apache. This setup also allows for multiple IP Addresses (for SSL Cert) and multiple Varnish Backends.
This article assumes you have working knowledge of Linux and CentOS distribution in particular. This installation is based upon the “yum” paths for all instances. It also assumes you have apache setup already, serving content with multiple IP Addresses and your server’s external IP Address 1 = 2.3.4.5 & IP Address 2 = 6.7.8.9
———————————————————————————————————
Pound SSL Wrapper
Pound is a reverse proxy, load balancer and SSL wrapper that allows you to listen on a secure port (443), decrypts the SSL Certificate and then sends the decrypted data to a local non-ssl port (80) to be served by the Varnish. First, let’s setup the pound configuration.
yum install pound
edit /etc/pound.cfg
ListenHTTP
Address 2.3.4.5
Port 80
Service
Redirect "https://www.mydomain.com"
End
End
ListenHTTPS
Address 2.3.4.5
Port 443
Cert "/var/certs/mydomain.com.pem"
Service
BackEnd
Address 127.0.0.3
Port 80
End
End
End
In this example, ListenHTTP, we want to make sure that the entire site is served securely and redirecting 80 traffic to 443. In ListenHTTPS, we pass it the path to the cert in .pem format. Not only is this required for ListenHTTPS, but it MUST be in .pem format.
To convert a crt to a pem, run the following command (replacing files with your own)
“cat mydomain.com.key mydomain.com.crt (mydomain.com.ca) > mydomain.com.pem”
Varnish Caching
Varnish is a Reverse Proxy (Caching, HTTP Accelerator) that allows for Load Balancing.
yum install varnish
edit /etc/sysconfig/varnish
I used Alternative 2 for the configuration. Set the IP(s) in the -a option. While the IPs are comma separated, make sure there isn’t a space between the IP addreses. It will fail. Pound is listening for the IP Address 2.3.4.5, so we don’t need to have Varnish listen for that IP. We need to have it listen on the port that pound specified as its backend. In this case, 127.0.0.3:80. Since pound is not serving secure content for IP Addresss 6.7.8.9, we can have varnish listen for that IP.
## Alternative 2, Configuration with VCL
DAEMON_OPTS="-a \"6.7.8.9:80,127.0.0.3:80\" \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-u varnish -g varnish \
-s file,/var/lib/varnish/varnish_storage.bin,1G"
Next, we need to edit the default vcl file for Varnish which is where all the settings will be added.
/etc/varnish/default.vcl
# For IP Address 6.7.8.9 and the rest of the domains on the server.
backend default {
.host = "127.0.0.2";
.port = "80";
}
# For IP Address 2.3.4.5 listening on Pound, proxied to varnish on 127.0.0.3. Setting Apache Backend to 127.0.0.4
backend mydomain{
.host = "127.0.0.4";
.port = "80";
}
sub vcl_fetch {
# Add a unique header containing the cache servers IP address:
remove obj.http.X-Varnish-IP;
set obj.http.X-Varnish-IP = server.ip;
}
sub vcl_recv {
# choose a backend depending on domain
if (req.http.host ~ "www.mydomain.com") {
set req.backend = mydomain;
}else{
set req.backend = default;
}
# remove and reset the clients IP address
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For=client.ip;
}
In the sub vcl_recv and sub vcl_fetch, I am doing two things. 1. Setting the backend server depending on which domain it is looking for. 2. Passing along the client’s IP Addresss to reflect in the apache log files.
SSL Conf
edit /etc/httpd/conf.d/ssl.conf
Pound is now listening on port 443, which means you can’t bind the ssl.conf to 443 as well. We need to comment out the listener.
# Listen 443
httpd.conf
/etc/httpd/conf/httpd.conf
We need to set apache to listen and set the NameVirtualHost on the ports that we specified in /etc/varnish/default.vcl
Listen 127.0.0.2:80
Listen 127.0.0.4:80
NameVirtualHost 127.0.0.2:80
NameVirtualHost 127.0.0.4:80
Vhosts File
/etc/httpd/sites-available/mydomain.com.conf
Finally we need to change the virtualhost IP Address in the vhost.conf to listen to the same ports as above depending on domain.
<virtualhost 127.0.0.2:80>
...
</virtualhost>
or
<virtualhost 127.0.0.4:80>
...
</virtualhost>
If you need the client’s IP Address (I don’t know why you wouldn’t), you need to add the following into the virtualhost above. Replace with your log file paths
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" varnishcombined
CustomLog /www/mydomain.com/logs/access.log varnishcombined
Now that we have finished the setup, this will require a restart of all services for it to work. If it was setup properly, they should all say “ok”.
/etc/init.d/pound start (restart)
/etc/init.d/varnish start (restart)
/etc/init.d/httpd restart
Monitoring Varnish Stats, Logs and Histogram
Based on install paths, the following should work. From the prompt: varnishstat, varnishlog, varnishhist
Varnishstat and Varnishlog give detailed information on headers being passed and the status of the cache. Varnishhist is a histogram of the traffic coming to your site. The left side is cached content and the right side is server generated content.
Stress Testing New Setup
If you have very little traffic, you might need to use a stress test tool to find out how the server will handle your new setup. Siege is a tool that forces many concurrent connections. If you run this tool before implementing Varnish, it may but more likely will bring down your server. The load average will shoot up very quickly (to 80.00+) until the server can’t handle it and stops responding. After installing Varnish, you can run up to thousands of concurrent connections and the load average won’t go past 10.00 (depending on many factors).
The parameters being passed to siege below, -c350 (350 is the number of concurrent connections, default is 15), -t1M (1M is the total time that siege will run for, 1 minute)
From command prompt, type:
siege -c350 -t1M www.mydomain.com
* It helps to have multiple shells up and watch “top”, “varnishstat” and “varnishhist”.
Log Files
tail -f /var/log/messages and /var/log/varnish/varnish.log
« Previous entries Next Page » Next Page »