June 25, 2011
Joomla, Varnish Caching, Cookies and Heavy Server Load
Recently, we (my company) took over web hosting for one of our clients. On average, they get 500,000 visits per month and a high profile suite of sites. I setup a Cloud Server Farm for the client. One of their sites, is the biggest of the group and their flagship site. This site was built 2 years ago using Joomla, a Content Management System, by a third-party company. I am not an advocate of Joomla because of the trouble that we have had with it. It is a very resource heavy CMS and for me, the problems outweigh the benefits.
While setting up this site, everything was working great until I started to test the server under high stress using “Siege”. After running the test with 50 requests/second, the server started to buckle and after 15 seconds, Apache became unavailable. I spent about 2 weeks trying to figure out what could cause this problem, but until today, I have been unsuccessful. I will first explain what I tried and how each change I made, either did nothing or caused different problems making them not a viable solution.
My first thought was to optimize Apache to handle more connections at one time. Apache’s HTTP Server website has a Performance Tuning section. After making these changes and restarting Apache, the Stress Test still caused the server to die.
Next, I thought it might have been the mysql server that was the issue. Even though, the mysql server wasn’t getting high load, I decided to fine-tune mysql as well. Since we have a dedicated mysql server, I used the configuration that was located in mysql-huge.cnf. This configuration is designed for servers that are just used for mysql. These changes seemed to help, but it wasn’t even close to what the site should be performing at.
I then thought it might be the amount of RAM on the server. I resized the cloud server to 4G of memory, but there was still no change. I resized it back down to 1G of memory which is what we thought it should be at.
After looking into more server configurations as the problem and getting nowhere, I decided that I would see if Joomla was causing the issues. You should know that this is all being done on a development server. I started by removing the Javascript files that were being included to display the client’s Ads. They have a total of 32 JavaScript calls and takes 10 seconds to just load the Ads. Removing them helped speed up the site, but the high load server failures were still happening and I added their Ads back in.
Now, the server is using Varnish for caching the content. It should be holding onto a static version of the site for 2 minutes (what it defaults to), but it wasn’t keeping the cache. Apache was still getting called every time you load the site. Varnish was still pulling the page through cache, but it was not keeping the cache. I have Varnish running on all of our servers including my personal server farm, however, I couldn’t figure out why this site was not keeping cache.
I tried removing some of the third-party modules that could have been incorrectly programmed and causing serious performance issues. After removing all of the modules on the homepage, the problems still existed. I got to the point where I didn’t think that it was anything to do with the server or how the site was pulling its content through Joomla, but I felt that there was something on the Joomla Core that was the problem.
I spent the next few hours researching what Joomla could be doing that was making the server die. After finding this blog article, I had an idea. His problem wasn’t my problem, but something that he wrote got me thinking. What if, the problem is with the Joomla Cookies. I did a few more Google searches and found that Joomla does not handle cookies very well it uses cookies for everything Joomla does.
*Please read the rest of this before trying the following as this configuration did not work even though it was performing the way I thought it should.
This site is on a dedicated server and no other traffic will be going to it. It is a 64-bit Quad Core Centos Server with 1G of ram. In Varnish, I added the following to the VLC file.
sub vcl_fetch {
set beresp.ttl = 5m; # Time that the cache should be kept for.
unset beresp.http.set-cookie; # Unset all cookies on the site
}
I “sieged” the server again at 50 requests/second and sure enough, Varnish was keeping the cache and serving it out of Varnish and not hitting apache at all. I then proceeded to stress the server at 500 requests/second and still the server was holding strong. Finally, I stressed it at 800 requests/second and it was still functioning properly. This is more than what the client would receive for traffic on an average day and I was confident that this had solved the problem….read on!
I then tried to log into Joomla on the front-end and on the administrative side. As it turns out, removing the cookies was a serious problem for the functionality of Joomla and would not let me login. I then removed the change that I made to the VLC, restarted Varnish and tried to login again. This time it worked.
I thought about how I could work around this issue, but nothing was coming to me. I did more research and came across this blog article. It is a little bit hard to read, but once you make the configuration readable, I found this code. *Note that I changed the obj.http.set-cookie to beresp.http.set-cookie to account for the new version of the Varnish 3.0 language.
sub vcl_fetch {
set beresp.ttl = 5m;
if( req.request != "POST" ) {
unset beresp.http.set-cookie;
}
}
Once I added this code to the VLC and restarted Varnish, not only could I login to Joomla on the front-end and administrative side, but the server was still holding strong at 800 requests/second.
Now, I can only stress the server to the point where I think it will hold its own in a real life scenario, but that doesn’t mean other steps may need to be implemented if the site gets picked up by Digg or Google (Google is what took down this clients server last month). Until that happens, we can’t account for the amount of traffic the site will receive. One additional step you can take is to add a Load Balancer into the mix and allow for the additional traffic and relive some of the stress on the sole server. This however, will be a separate blog entry as this one is already long enough.
In closing, I hope this article helps you to save weeks of performance testing, server changes and headaches.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_c.png?x-id=f633c631-91ff-4ade-9479-844a3af0647a)