Thursday, March 17, 2005

Adventures with Apache 2

I've been again spending some quality time with Apache 2. The Apache Group has been stating that we consider this release to be the best version of Apache available and encourage users of all prior versions to upgrade for quite some time now; you can find this in official announcement and here and there elsewhere. Apache 2.0 does offer some major improvements Apache 1.3 (like filtering and internationalized error messages). It is also significantly, measurably faster.

Besides the comments by the Apache Group, you'll also find articles and discussions questioning whether or not to upgrade and often implying that upgrading is a bad idea. The usual conclusion they come to as to why one may not want to upgrade is that not all third-party plug-ins have been upgraded to 2.0 yet. This is of course becoming less and less of a problem as time goes on, and is thus becoming a flimsier and flimsier argument.

Still, of all the production servers I manage, only one of them is currently running Apache 2. I'd love to upgrade the others, for the speed increase if nothing else, but I am still prevented from doing so. My problem isn't third-party plug-ins (in fact, the three plug-ins I most frequently use in Apache 1.3 are now standard components of Apache 2.0, so the plug-ins are actually easier with 2.0) but is instead something I haven't yet seen mentioned as a deal-breaker with Apache 2.0: caching.

With Apache 1.3 caching is robust and reliable. With Apache 2.0 caching is something of an adventure. I'd say it's a little more hit or miss, but actually it even lacks the X-Cache HIT or MISS HTTP header lines that make checking caching performance on Apache 1.3 so easy.

Going through the documentation for the Apache 2.0 versions of modules mod_cache and mod_disk_cache, one finds a lot of messages stating that various directives are not yet implemented, and that in fact both modules are still experimental. They're also (often cryptically) buggy.

You might wonder why I need the Apache caching facility on the majority of my servers. The answer is simple: Zope. (where I work days) is a Zope Solution Provider, and the majority of sites we serve utilize the dynamic capabilities of Zope behind a robust Apache front-end, and Apache caching is a simple and effective way of reducing a Zope server's load.

Every so often I try Apache 2 on a non-production machine to see how it handles caching, and each time I come to the same basic conclusion that it's not yet ready for prime time. Well, that is until 2.0.53 came out on February 8.

I tried it as usual and had it running for quite awhile on a test machine without problems. The lack of the X-Cache header made testing more painful than it should otherwise have to be, but it was still readily possible to tell whether a request had been served by Apache via cache or by Zope via dynamic generation. I actually started to dig into the source at one point to see if I could temporarily reactivate the header for testing purposes, but doing so was non-trivial and I did have other deadlines. In any case it looked pretty good even in a relatively complicaated fake set-up.

I then tried it on our ShellTown machine. This machine primarily exists to provide shell accounts for customers; it only serves a handful of domains, basically just for itself and the shell customers who utilize it.

Things seemed fine at first, but after a couple days we got some trouble reports in that involved a very strange behavior: some of the domains would work properly if addressed without a prepended www. but wouldn't work properly in the opposite case. Specifically for the most troublesome case, the site Ochmoneks worked reliably 100% of the time if it were accessed as but would produce a virtual directory listing (which even showed the proper index.html file in it) if it were accessed as Weirdly, one could make it work if two trailing slashes were appended (one wasn't enough). Also, of course I checked the DNS records for, and they were fine and showed nothing unusual.

Turning caching off completely made the problem go away. Doing a search through the Apache Software Foundation Bugzilla reveals nothing that sounds similar out of the over fifty reported bugs involving caching in Apache 2.0, although I confess I've not been able to reduce this to a simple case. It only seems to happen on the real server.

The Ochmoneks site doesn't utilize Zope, so my workaround has been to hand-tune the httpd.conf file to cache only Zope-generated content. This is far from ideal, but there are few enough domains (and few enough Zope-generated pages as some domains feature both static and dynamic content) to make it viable.

Thus the ShellTown customers get the advantages of Apache 2.0, but due to the oddities we've experienced we're in no hurry to upgrade any of the more complicated machines.