NIX/WIN/WEB

* Benchmarks current as as of 2.17.10

Who’s the fastest: Apache on Windows or Apache on Linux?. In the course of deploying a new web server I thought I’d find out. I play around with APC, Database tuning, and so on. Among other things, in the end we see how with a few simple settings we can increase our servers capacity 2 times over.

For these tests I use Apache Bench (ab):

-On Ubuntu it’s in usr/sbin/ (if you’ve installed the apache2-utils package)

-On Windows in %path_to_apache%/Apache2/bin/ab.exe

I started out with a Zend Server CE installation on the Windows boxes, then moved to a custom build, which is actually just the individual downloads of Apache and PHP.

Our test systems:

Server 2008 x4

2.66 GHz Dual-Core Xeon – PHP 5.2.11-ts – 6 GB RAM – Zend Server CE

Server 2008 x4 C

2.66 GHz Dual-Core Xeon – PHP 5.2.11-ts – 6 GB RAM – (Custom)

Server 2008 x4 C + APC

2.66 GHz Dual-Core Xeon – PHP 5.2.11-ts – 6 GB RAM – (Custom)

Server 2008 x8

2 x 2.80 GHz Dual-Core Xeon (8 core) – PHP 5.2.11-ts – 6 GB RAM – (Custom)

Database Server

2.66 GHz Dual-Core Xeon – MAMP Stack / OSX 10.6.2 / 6 GB RAM

Ubuntu x2 +APC

2.20 GHz Pentium Dual E2200 – PHP 5.2.10 – Ubuntu 9.10 PHP / APC 3.0.19

Ubuntu x8 +APC

2 x 2.80 GHz Dual-Core Xeon (8 core) – PHP 5.2.10 – Ubuntu 9.10 PHP / APC 3.0.19

** Apache was configured the same for Windows and Ubuntu: for example, ThreadsPerChild is the same, same extensions (mod_rewrite) etc. However, as this is Windows, a Forefront client is running on the test server.

Generic benchmark conclusions:

If you’re running a Linux PHP web server you need APC. End of story. While it doesn’t always make your requests faster, it cuts RAM in half and CPU usage by a 1/4 or more in most cases. Bottom line your server simply scales more reasonably with APC.

On Windows it routinely doubles the served pages per second, which alone makes it an easy call. However, as we’ll see shortly I had trouble getting APC to work with Joomla, which could spell trouble for your site as well. This makes it a tough call—the best advice is to try before you buy (though it’s free, just to be clear).

Finally, hardware clearly makes a difference: the more cores you have the better. Both Ubuntu and Windows Server 2008 scale in a simple way: double the cores means double the requests served. Toss in APC and your basically getting extra web server capacity for free.

Real world tests.

For the next test we move away from simple pages to ones with database calls. I simply installed Joomla along with the sample data set and hit the index page.

Default Joomla Install Page: index.php / 1000 Requests / 100 Users

I wasn’t happy with this performance, and soon noticed my database server (quad-core 2.66 / MySQL 5.1.37 / 6 gigs), was saturated (400% cpu) when running the Ubuntu 8x + APC combo.

Thus, we now tune the MySQL database to use query cache (it’s disabled by default), as well as have larger buffers for results and sorting.

With our tuned database, for the next test we disable APC on the Ubuntu box and install 6 gigs of ram (we upgrade to the PAE Linux kernel to allow for more than 2 gigs of RAM). We were not swapping before, but we almost certainly will now. As none of the other tests had swapping issues, we keep things fair by upping the RAM to be safe.

Ubuntu x8 6GB RAM            (43.57) Requests/Second ~120%-300% db CPU

The results are close to what we had before, because at this point 45 requests/second seems to be the saturation point of the web server in terms of CPU. Now, with APC enabled again:

ab -n 1000 -c 100 http://localhost/joomla/index.php

Ubuntu x8 6GB RAM            (78.88/57.44) Requests/Second

Much better–CPU on the web server pegs at times, but the CPU graph looks more like a roller coaster than a flat road, with regular dips into 50% . What’s more, with the query cache MySQL usage is around 250%, not the 400% it was before. Also, RAM on the web server server is slashed by almost a half with APC enabled: (2.4 GiB vs. 1.3 GiB)

Thus, with APC and enabling the query cache we removed the MySQL bottleneck and allowed the requests per second to go much higher.

Let’s try one more test with 200 concurrent users instead of 100:

ab -n 1000 -c 200 http://localhost/joomla/index.php

Ubuntu x8 6GB RAM            (74.09) Requests/Second

So our server is holding steady with more concurrent users.

As far as database, our Query Cache Hit rate is right around a perfect 70%

http://dev.mysql.com/tech-resources/articles/mysql-query-cache.html

However, I notice that after a few more runs our requests/second start to drop into the 40′s. A quick check of our APC stats shows plenty of free memory (apc.shm_size = 64 / 58 free), so I suspect the MySQL Query Cache. I originally set the query cache to 64 MB, but after a few million queries we’re down to 12 MB. However, we don’t want the cache too big, as performance will start to suffer when it has to sort through all the cached data.

I issue a: FLUSH QUERY CACHE command to defragment the query cache. Technically this doesn’t clear the queries in the cache, nor does it free the memory those queries consume, it just performs a memory defrag.

ab -n 1000 -c 200 http://localhost/joomla/index.php

Ubuntu x8 6GB RAM            (32.41) Requests/Second

Nothing. Thus, I’ll issue the: RESET QUERY CACHE to clear it completely and run the same test again:

Ubuntu x8 6GB RAM            (41.26) Requests/Second

Where did our performance go?

Now of course before I get too far let’s just say that 36/requests a second is by no means terrible: that’s around 129,00 views an hour, 3.1 million a day, or 96 million a month—on one server.

But we’re talking cost of equipment here, so I still want my performance back.

I do a full reboot on the web server, which as far as PHP is concerned, should be no different than when I manually reset the APC caches. No dice. We still get around 30 queries a second. What about a database server restart? Nothing. Still 30 queries a second. So even with all cache’s cleaned an cleared we still dropped by more than 50%

At this point I’m at a total lose when it suddenly hits me…what about the Joomla database? When doing some of my tests I’ve been using the free MySQL GUI tool MySQL Administrator to peek at what queries are being run. The first thing that jumped out at me is the sheer number of queries to a table called jos_session. I launched MySQL Query Browser and took a look at the table: nothing special: about 9000 rows with 20 MB of data.

Still at a lose, and because I imagine this session data is volatile I issue:

DELETE FROM jos_session

And run the benchmark again:

Ubuntu x8 6GB RAM            (74.68) Requests/Second

There we go, back to normal.

What’s more, as expected after about 7 runs the requests per second begins a downward trend—this seems to happen as each run of 1000 requests adds about 1.5 MB to the size of the jos_session. All told, after we hit the 10 MB mark for table size the requests are well into the 40′s.

So how do we fix this?

I suspected and soon confirmed that Joomla is issuing a host of costly quires to this table. For example, one query:

SELCT guest, usertype, client_id FROM jos_session WHERE client_id = 0

Seems innocent until you see that as every anonymous user has a client id of 0; so our query always returns the total number of rows in the jos_session. After 10 runs we have close to 11,000 rows being returned on every call.

The good news seems to be that Joomla removes rows from this table after a set time. Of course my benchmarking is placing a strain on the system, but not much more than a really popular site would have.

No matter, the problem then is that this table becomes a major bottleneck, one that potentially halves the performance of you box should the traffic become high enough. I’ll leave it at that for now as I’m not about to tweak Joomla, but it’s an interesting find.

I ran the Joomla test again after a fresh clearing of the jos_session table for select configs:

Unfortunately I could not get APC to work with Joomla on Windows, but when I do will report back.

Conclusions

When I first started these tests nothing was optimized. The first lesson then is that out of the box none of the standard solutions are suited for high-demand production servers. You simply must tune and optimize. The one exception to this is a Zend Server CE stack, as while it wasn’t as fast for very simple PHP pages, it was much better for anything you’re actually likely to have on a real website. In short, while Zend Server CE’s Optimizer+ isn’t actually supposed to make your PHP code faster, it sure does in some cases. Of course should you want a fully optimized Zend stack out of the box, Zend Platform is what you’d want on Windows, or any other that includes op code optimizations out of the box.

Taking a quick overview then, my tests are composed of a decent spectrum of uses. From static HTML and simple PHP, more complex PHP, all the way to a full blown CMS.

On the static content side it hard not to be impressed by the incredible .html output of Apache on Linux. A dual core Pentium is 3x faster than a Server 2008 8 core Xeon box. At over 14k requests/second, if it’s static delivery your after look no further.

On The MySQL Query Cache

It’s hard to argue against this one. The biggest savings that’s I’ve seen are that when enabled, you’ll drop your CPU load by a significant amount. Typical gains I’ve seen when running the Ubuntu 4x benchmark are CPU load gong from 11-14% down to 6-8%.

More importantly though, as in the example above we are going from an average of 14% down to 8% utilization. This means that while on average you’re not going to get a higher query-per-second rate if the web server is already pegged, you will be able to serve more request if needed. This will come in handy if you database and web server are on the same box (which they will be the case for 90% of regular users on hosting plans)

I should also say that the MySQL query cache is going to be far less effective in situations where the cached query data changes often–and yet this is exactly what happens with that costly jos_session table. In other words, even in a situation where your tables are not perfectly suited for a caching solution on paper, you’re still bound to benefit greatly by enabling the MySQL query cache.

On Windows and PHP Performance

On the most general level it’s an easy call: unless you need IIS or MSSQL their is no reason to host PHP on Windows. Of course to be fair Microsoft has made great strides with FastCGI, and their is every possibility that PHP/FastCGI numbers could match or even best what we’ve seen here today: but when you factor in licensing costs and compatibility issues (good luck running APC or handy extensions like imagick), why go with anything else?

The bottom line is that Apache/PHP on Ubuntu is nearly twice as fast as it is on the same hardware on Windows Server 2008—when you toss in APC, again, of which I could only do on Linux, it’s nearly 4 times faster.

Cost and compatibility aside, to be fair, on the performance front this is most likely more an indictment of Apache on Windows than Windows. You can bet that static content delivery on IIS is also blazingly fast.

Also, the simple fact is if you’re a developer Windows is always a solid choice, as any of the configurations covered today would be more than adequate for your development environment. FormBoss is developed entirely in Windows and I couldn’t be happier—I just wouldn’t host it on Windows is all.

On Linux and PHP Performance

It’s hard not to be struck by the little Ubuntu box. This isn’t even server software here, just a plain jane Ubuntu 32-bit Live CD (and probably not even i686). I’m sure their is more performance to be squeezed from the build, but it’s already so far ahead of the pack that we’ll just leave good enough alone for now.

Perhaps most surprising is that, along with the static HTML performance, just how much faster it is on the 8 core Xeon with the Joomla test—a 4x increase on the same hardware is simply stunning. That said, as the Ubuntu scores on the much slower dual-core Intel shows, you can’t skimp that much; you’ll still need something decent with regard to CPU and RAM if you plan on hosting lots of traffic.

The Most Important Part…

The most important lesson is that no matter you do with your hardware, it the software that’s gonna get you. Our example with Joomla may not necessarily be because of bad or sloppy code, but the server won’t care. It’s up to you to design and deploy well written code and infrastructure for your site. Not doing so may end up costing you dearly in the end.

Finally, On The 8 Core Xeon

What…a…beast. I still can’t believe the numbers, but not only does it scale with more cores, because of memory bandwidth, bus, and who knows what else, it scales better than a simple ratio of cores vs. performance.

That thing simply screams and I love it. For the record, it’s a Mac Pro Gen 2.