<<< Timothy W Macinta's Website Menu >>> Home | My Java | Resume/CV | R&D | Age Detector | Contact

Lighttpd Patches

In March, 2005 I discovered a fantastic new web server named lighttpd. It caught my interest because it scaled much better than Apache for what I needed, handling more traffic with several orders of magnitude less load. It was also substantially more feature rich (e.g., it supported PHP) than other scalable web servers like Boa and thttpd and also appeared to be more actively maintained. I set about to switch some high traffic servers that I provide architectural support for to lighttpd and in the process I created some modifications to the server to optimize it for the specifics of the system.

Efficiently serving large files from PHP/fastcgi/etc.

Note: the patch below has been accepted by the lighttpd maintainers and has been integrated into the official release starting with version 1.4.4. The details for using it are slightly different, so please refer to the latest lighttpd documentation and source code for the most up to date usage information. I am keeping the original instructions intact below for historical purposes.

The system that I was in charge of architecting and implementing needed to serve a large number of large files to a large number of concurrent users, but with flexible access restrictions. The flexible access restrictions would require the use of some server side scripting (using PHP in this case given that it was in use elsewhere in the system), but the serving of the actual file would be better off handled by lighttpd itself because lighttpd has some inherent efficiencies when serving static files that would be lost if the files were to be piped through a script. Lighttpd comes with a module called 'mod_secure_download', which is a great fit for this sort of use.

I created a patch to take the concept one step further. My patch allows dynamic scripts to issue essentially an internal redirect to lighttpd so that it will serve a static file as the response body. This would have the benefit of allowing a dynamic script to locate and authorize the content while taking advantage of the efficiencies lighttpd achieves when serving large files. This provides the same benefits as mod_secure_download without the need to send a redirect to the user and with more flexibility in the location of the served content (the content for mod_secure_download needs to be below a particular directory and the sub-path is exposed as part of the URL).

I am making my patch available here in case anybody else would have use for it. Currently, it's just a modification for the fastcgi module in the version 1.4.0 pre-release 1. I figured I would hold off on making it apply to all modules until the output filters are added. Plus this covers PHP by virtue of being used via fastcgi, and that was what I needed.

Before using this patch, please consider the security implications for your server. This patch will allow your dynamic pages to serve any file that lighttpd has access to. This may be undesirable if you are providing virtual hosting and you confine the fastcgi scripts for different users to run under the UID of the applicable user. I would only recommend using the patch on a server where the content is fully trusted. In any case, there is no warranty.

To use the feature, have your script set a response header like the following:

    X-send-response-body: /usr/src/linux-2.4/README

The header value should be the full path to the file that you want to serve. The patched fastcgi module will recognize the header, replace the value for security purposes, and then send the given file as the response body.

I also like to set the environment variable "X_SEND_RESPONSE_BODY_SUPPORTED" in my "lighttpd.conf" settings for what gets passed to PHP so that my scripts know whether or not they are running under a patched server that supports this, and if not to fall back on serving the file manually. That isn't essential, though.

I've tested the patch under several months of very heavy load on multiple servers and it seems to work well. Here are the relevant files:

• Original mod_fastcgi.c from lighttpd-1.4.0-20050402-1520
• Modified mod_fastcgi.c with my patch
• A patch file, if you prefer
• The lighttpd license enclosed with this version