Tomcat improvements coming up in 10.04 !

Jason Brittain, a software architect at MuleSoft and the author of Tomcat: the Definitive Guide, approached us with some ideas about how to improve the Tomcat package in Debian and Ubuntu. Check out his blog post on working with Debian and Ubuntu here.

Jason noticed that the Ubuntu and Debian init scripts were starting Tomcat via the JSVC service runner, in order to allow binding the Tomcat JVM to privileged server port numbers (port numbers lower than 1024), while still running as an unprivileged user. However, JSVC was also the source of several long-standing bugs: it shuts down Tomcat abruptly and implements unreliable restarts, so it was possible for the init script to be unable to restart Tomcat. Upstream recommends using Tomcat’s catalina.sh script instead, and that script should in turn run the Java binary, so that the init script can properly handle any issues with starts, stops, or restarts. But that script doesn’t have the support to bind ports as root and then run as an unprivileged user, like JSVC does…

So, without JSVC, the Debian/Ubuntu Tomcat 6 init script would need to use some other mechanism to allow Tomcat to bind to privileged ports. As a replacement for binding via JSVC, Jason proposed using either Linux Capabilities (the setcap command), or the authbind package.

The setcap command did not work because of a JVM bug, so Jason used authbind. Authbind is written specifically to allow a process that is not running as root to bind to one or more privileged ports. It turned out to work well, and that’s now what the tomcat6 package uses.

Here are the changes that are implemented:

• JSVC is no longer used by the package. Instead, Tomcat is invoked via the stock catalina.sh script. Any script code in the init script that was used to generate the Java startup arguments has been removed because catalina.sh performs that work.
• Authbind is now the standard method for binding Tomcat to ports lower than 1024. There is a new option (disabled by default) to enable authbind in /etc/default/tomcat6. Enabling authbind means that Tomcat will be allowed to bind to privileged ports, if Tomcat or its webapps are configured to do so, and if authbind is configured to allow it. Enabling authbind also presumes the use of IPv4, since authbind only works with IPv4. The permission for Tomcat to use privileged port numbers is now configured at Tomcat package install time. Tomcat itself can be run entirely as an unprivileged user for this reason, tightening the scope of escalated privileges, improving server security.
• The security manager now defaults to the disabled state. It is commented that way in /etc/default/tomcat6. Having it disabled by default (but still supported in the code) means that fewer Tomcat package users will have difficulty getting Tomcat working in the default configuration. It is also consistent with the default upstream distribution settings.
• Reliable restarts are now implemented in the init script. A stop command sends a SIGTERM to the Tomcat JVM, running Tomcat’s shutdown hook. The init script monitors the process after sending this signal and will exit right away when Tomcat’s JVM exits. If Tomcat’s JVM does not exit, the init script waits for 20 seconds, monitoring the JVM process. At the end of the 20 seconds, the init script sends a SIGKILL to the JVM and waits up to 5 additional seconds for the JVM to quit. If the JVM process will not quit (kernel I/O hang problem, or similar), the init script prints a failure message saying that Tomcat will not shut down. Tomcat can now be reliably and gracefully restarted via the init script. Graceful shutdowns mean that no webapp data is lost due to the Tomcat JVM being forcefully killed. Reliable restarts means that it is sufficient to script a “service tomcat6 restart” command, and you can be confident that the currently running Tomcat will be shut down gracefully and that a new Tomcat process will be started only after the first one is gone. Implementing graceful stops means that webapps have the appropriate amount of time to persist their application data to disk before the Tomcat JVM exits.
• Tomcat now sends STDOUT and STDERR to its usual stock log file, $CATALINA_BASE/logs/catalina.out (for example, /var/log/tomcat6/catalina.out in this package) alongside all of the other Tomcat log files, instead of logging STDOUT and STDERR to syslog.
• The tomcat6 package is now up to date with the latest stable release of Tomcat, 6.0.24. This fixes a number of important issues, as Jason described in a recent blog entry.

This was a great example of collaboration between distributions and someone close to the upstream project. We chose to implement and test the solution in Debian and sync the Ubuntu version to the Debian one. All these improvements have been implemented in time for next week’s Ubuntu 10.04 LTS and the future Debian Squeeze (6.0) releases.

The collaboration effort between MuleSoft, Debian and Ubuntu significantly improved the reliability and usability of our tomcat6 packages, and we’re very happy to have Jason contributing to our community. Many thanks as well to Niels Thykier, Ludovic Claude and Torsten Werner on the Debian side for making it all possible !