Background
I’m beginning to venture into things like logstash and puppet at work to try to sanitise the administration and logging of an ever-growing number of servers and applications.
Getting logstash going is simple if you know how, and non-trivial to do it in a managed environment such as ours using Spacewalk.
Getting rsyslogd to feed into logstash is also simple enough, but then having said that one of the next things on my list is dumping the standard “syslog” input for the tcp+grok version on the logstash cookbook due to the limitations of the syslog input.
The one that will screw you into the ground is getting tomcat-6.0.24 to push all its logging into logstash. The main reason is because the version of tomcat6 available currently on CentOS6 (6.0.24) is hard-coded to use java.util.logging for its core logging, and the format is horrific. The datestamp doesn’t even come until the second line of the entry. Tomcat does now provide a OneLineFormatter to try to make it a little more sane, but it’s not available until tomcat-6.0.32 leaving those on standard CentOS6 set-ups SOL.
UPDATE 02/12/2013: As of CentOS 6.5 you should no longer require the first step of replacing the hard-coded JULI JAR files (tomcat-juli.jar and tomcat-juli-adapters.jar) with fresh JCL copies. This has now been done upstream: http://rhn.redhat.com/errata/RHBA-2013-1721.html
But it IS solvable.
- Configure tomcat6 to use log4j instead of java.util.logging (Prior to CentOS 6.5)
- Configure log4j to output in a logstash-friendly JSON format
- Use your favourite shipper to get ship the output to logstash
Note, throughout this I try to use the full true paths to files rather than symlink locations to save confusion
Getting the Job Done
Configure tomcat6 to use log4j
tomcat6 uses JULI as a kind of wrapper to write out via java.util.logging. Not content with using the Java Commons Library (JCL) version, it ships with it’s own version that is hard-coded to the defaults. In order to change the defaults, you have to replace JULI with the JCL version.
rm /etc/tomcat6/logging.properties
# These two lines iff using CentOS6 prior to v6.5
wget -O /usr/share/tomcat6/bin/tomcat-juli.jar "http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.24/bin/extras/tomcat-juli.jar"
wget -O /usr/share/java/tomcat6/tomcat-juli-adapters.jar "http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.24/bin/extras/tomcat-juli-adapters.jar"
# Carry on
wget -O /usr/share/java/log4j-1.2.17.jar "http://archive.apache.org/dist/logging/log4j/1.2.17/log4j-1.2.17.jar"
ln -snf /usr/share/java/log4j{-1.2.17,}.jar
If you wanted to just stop there and have the default logging replaced by log4j without JSON output, create /usr/share/java/tomcat6/log4j.properties (taken from http://tomcat.apache.org/tomcat-6.0-doc/logging.html#Using_Log4j):
log4j.rootLogger=INFO, CATALINA
# Define all the appenders
log4j.appender.CATALINA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.CATALINA.File=${catalina.base}/logs/catalina.
log4j.appender.CATALINA.Append=true
log4j.appender.CATALINA.Encoding=UTF-8
# Roll-over the log once per day
log4j.appender.CATALINA.DatePattern='.'yyyy-MM-dd'.log'
log4j.appender.CATALINA.layout = org.apache.log4j.PatternLayout
log4j.appender.CATALINA.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
log4j.appender.LOCALHOST=org.apache.log4j.DailyRollingFileAppender
log4j.appender.LOCALHOST.File=${catalina.base}/logs/localhost.
log4j.appender.LOCALHOST.Append=true
log4j.appender.LOCALHOST.Encoding=UTF-8
log4j.appender.LOCALHOST.DatePattern='.'yyyy-MM-dd'.log'
log4j.appender.LOCALHOST.layout = org.apache.log4j.PatternLayout
log4j.appender.LOCALHOST.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
log4j.appender.MANAGER=org.apache.log4j.DailyRollingFileAppender
log4j.appender.MANAGER.File=${catalina.base}/logs/manager.
log4j.appender.MANAGER.Append=true
log4j.appender.MANAGER.Encoding=UTF-8
log4j.appender.MANAGER.DatePattern='.'yyyy-MM-dd'.log'
log4j.appender.MANAGER.layout = org.apache.log4j.PatternLayout
log4j.appender.MANAGER.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
log4j.appender.HOST-MANAGER=org.apache.log4j.DailyRollingFileAppender
log4j.appender.HOST-MANAGER.File=${catalina.base}/logs/host-manager.
log4j.appender.HOST-MANAGER.Append=true
log4j.appender.HOST-MANAGER.Encoding=UTF-8
log4j.appender.HOST-MANAGER.DatePattern='.'yyyy-MM-dd'.log'
log4j.appender.HOST-MANAGER.layout = org.apache.log4j.PatternLayout
log4j.appender.HOST-MANAGER.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Encoding=UTF-8
log4j.appender.CONSOLE.layout = org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
# Configure which loggers log to which appenders
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost]=INFO, LOCALHOST
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager]= INFO, MANAGER
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager]= INFO, HOST-MANAGER
Configure logging for JSON output
Get log4j-jsonevent-layout
Not as trivial a step as you would think.
If you’re using logstash 1.1.x, then it is just a case of getting the jar and the dependency jars, however I’m going to assume you’re a sane person and are using the latest logstash (currently 1.2.1).
Due to a change of format, you need to use a more recent log4j-jsonevent-layout than the 1.4 release. This is because of the new JSONEventLayoutV1 class. Don’t let this confuse you. It’s not how it seems that JSONEventLayoutV1 is the old version and JSONEventLayout is the new one, it’s the other way around. Think of JSONEventLayout as if it were JSONEventLayoutV0 and JSONEventLayoutV1 is the newer one that you need (that isn’t in v1.4).
At the same time, having looked at the fork tree, for the moment I have decided to use Spredzy’s master instead of the original logstash version, hoping they will merge later on.
This does of course mean you’re going to need to build the 1.5-SNAPSHOT yourself with maven or Jenkins.. oh wait, no you’re not!
wget -O /usr/share/java/tomcat6/jsonevent-layout-1.5-SNAPSHOT.jar "http://blog.tpa.me.uk/wp-content/uploads/2013/10/jsonevent-layout-1.5-SNAPSHOT.jar"
Get the other three dependencies
log4j-jsonevent-layout depends upon: commons-lang-2.4, json-smart-1.1.1 & junit-4.8.1:
cd /usr/share/java/tomcat6
wget "http://repo1.maven.org/maven2/commons-lang/commons-lang/2.4/commons-lang-2.4.jar"
wget "http://repo1.maven.org/maven2/net/minidev/json-smart/1.1.1/json-smart-1.1.1.jar"
wget "http://repo1.maven.org/maven2/junit/junit/4.8.1/junit-4.8.1.jar"
Configure an appropriate log4j.properties
Create or modify (from earlier) /usr/share/java/tomcat6/log4j.properties to reference the JSONEventLayoutV1 class.
log4j.rootLogger=INFO, CATALINA
log4j.appender.CATALINA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.CATALINA.File=${catalina.base}/logs/catalina
log4j.appender.CATALINA.Append=true
log4j.appender.CATALINA.Encoding=UTF-8
log4j.appender.CATALINA.DatePattern='.'yyyy-MM-dd'.log'
log4j.appender.CATALINA.layout = net.logstash.log4j.JSONEventLayoutV1
log4j.appender.LOCALHOST=org.apache.log4j.DailyRollingFileAppender
log4j.appender.LOCALHOST.File=${catalina.base}/logs/localhost
log4j.appender.LOCALHOST.Append=true
log4j.appender.LOCALHOST.Encoding=UTF-8
log4j.appender.LOCALHOST.DatePattern='.'yyyy-MM-dd'.log'
log4j.appender.LOCALHOST.layout = org.apache.log4j.PatternLayout
log4j.appender.LOCALHOST.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost]=INFO, LOCALHOST
And after a tomcat restart you should find /var/log/tomcat6/catalina happily spitting out JSON format log entries. Keep an eye on the catalina.out though for anything strangely still passing through java.util.logging.
Configure your shipper
In my case, I’m using logstash as the shipper, so my /etc/logstash/conf.d/shipper.conf looks something like this:
input {
file {
type => "tomcat"
path => [ "/var/log/tomcat6/catalina" ]
exclude => [ "*.gz" ]
codec => json
sincedb_path => ["/opt/logstash/.sincedb_catalina"]
}
}
output {
redis {
host => "<redis_host>"
data_type => "list"
key => "logstash"
codec => json
}
}
Works for me, your results may vary and all that malarkey. Have fun!
Recent Comments