Graylog2: Java, Ruby, MongoDB-powered log management, monitoring, and alerting

For developers, application logs are critical to figuring out what’s going on inside the apps we create. We tail them. We search them. We analyze and graph them. Graylog2 a slick log management, monitoring, and alerting tool powered by Java, Ruby, and MongoDB, performs these well. Graylog consists of a Java server that collects your logging data and stuffs it into MongoDB and a Ruby on Rails web interface for searching, filtering, and graphing that data.

Collecting: Graylog Server

Graylog’s server component requires Mongo version 1.6 or later and a Java environment.

Check out the project’s wiki for installation and startup instructions. Graylog also supports AMQP as an alternate transport for messages, just configure appropriately in your config file.

Graylog supports writing custom rules to determine what messages find their way to MongoDB and in what form, using Drools Expert, as in this example:

import org.graylog2.messagehandlers.gelf.GELFMessage

rule "Rewrite localhost host"
    when
        m : GELFMessage( host == "localhost" && version == "1.0" )
    then 
        m.setHost( "localhost.example.com" );
        System.out.println( "[Overwrite localhost rule fired] : " + m.toString() );
end

rule "Drop UDP and ICMP Traffic from firewall"
    when
        m : GELFMessage( fullMessage matches "(?i).*(ICMP|UDP) Packet(.|n|r)*" && host == "firewall" )
    then
        m.setFilterOut(true);
        System.out.println("[Drop all syslog ICMP and UDP traffic] : " + m.toString() );
end

Transport format: GELF

In addition to syslog format, Graylog also supports GELF, or the Graylog Extended Log Format, which offers

more than the 1024 bytes offered by syslog to accomodate more info such as backtraces
structured data.

Here’s a quick example of a GELF message:

{
  "version": "1.0",
  "host": "www1",
  "short_message": "Short message",
  "full_message": "Backtrace herennmore stuff",
  "timestamp": 1291899928,
  "level": 1,
  "facility": "payment-backend",
  "file": "/var/www/somefile.rb",
  "line": 356,
  "_user_id": 42,
  "_something_else": "foo"
}

A GELF message is just a GZIP’d or ZLIB’d JSON string. Check the GELF Spec for a list of required fields.

Searching and analyzing: Graylog Web Interface

Graylog also ships with a rather slick web interface for searching and viewing Graylog messages. Filters can be applied and saved into logical “streams”, allowing you to look at a slice of your data.

Graylog can even alert you when certain thresholds are exceeded for a given stream, as in this example email alert:

From: graylog2@example.org
To: lennart@socketfeed.com
Subject: [graylog2] Stream alarm! (Stream: Finance)
# Stream >Finance< has 23 new messages in the last 15 minutes. Limit: 15
# Description: Just a dummy stream with a not-so-random name but random data.

From: graylog2@example.org
To: lennart@socketfeed.com
Subject: [graylog2] Subscription (Stream: Finance)
# Stream >Finance< has 24 new messages since 2011-01-08 20:31:13 +0100
2011-01-08 21:12:38 +0100 from >localhost.localdomain<
  sundaysister kernel: [92837.097110] CPU0: Core temperature/speed normal
2011-01-08 21:12:38 +0100 from >localhost.localdomain<
  sundaysister kernel: [92837.096461] CPU0: Core temperature above threshold, cpu clock throttled (total events = 1485916)

Bonus: Capture exceptions with Rack

For Ruby web applications, there is a bonus Rack application that allows you to send all application exceptions to Graylog in the spirit of Hoptoad.

[Source on GitHub] [Web site]