Posted by Nathan Kaiser on Mon Jun 15 16:32:00 UTC 2009
Unified rails logging is an incredibly helpful tool, and isn’t that difficult to accomplish. Unfortunately, searches for the term come up mostly blank. Tim Lucas wrote a great article on using Eric Hodel’s SyslogLogger, but what’s missing are instructions on how to unify those logs on a central logging server. Don’t fear, this is a rather simple process with syslog-ng!
Syslog-ng is the “next generation” syslog drop in. It provides a much more advanced feature set than the default system syslog, and allows you to unify multiple log sources into single file with simple to manage directories. The configuration for syslog-ng can be a bit confusing at first, so we thought we would take the opportunity to document this setup from one of our customers installations. Hopefully this will help others create unified rails logs.
My Assumptions:This article is written with a few assumptions. First, I’m assuming you’re running a rails application that has multiple application servers. This could be both a “staging” and “production” server, or multiple production servers. Second, I’m assuming you’ve got a dedicated host you want to use as the receiving server. We normally recommend this to be a specialty slice and often, a customer will put this on their repository server.
The final setup will have your Rails application logging to the local syslog-ng setup on the same server as the rails application. That local syslog-ng will then replicate the logs out to the remote syslog-ng server which will collect and sort them into a shared log file.
These steps should be fairly straight forward, but if you have any questions, please don’t hesitate to contact us at support@blueboxgrp.com!
Step 1: Install syslog-ng on your servers.If you do your rails hosting with Blue Box Group, then you’re most likely running CentOS. If that’s the case, you can install syslog-ng with a simple yum command. You’ll then want to set it to start on boot!
[root@ staging-app01 /]# yum install syslog-ng - snip - Installed: syslog-ng.x86_64 0:2.1.4-1.el5 Complete! [root@staging-app01 /]# /sbin/chkconfig syslog-ng on
This process should be completed on all your application servers, as well as on the server you’re going to use as the central log box.
If you’re using a different distribution, you’ll want to use the appropriate tools (apt-get, etc) to install the syslog-ng package.
Step 2: Configure syslog-ngThe next step is to configure syslog-ng on each machine. As stated in the assumptions, there are two different types of configurations you need to worry about. The first configuration runs on your application servers and is what your Rails application talks to. The second runs on the collection server. I’ve laid out the configuration for both below.
Note: On CentOS, these configs are located in /etc/syslog-ng/syslog-ng.conf.
Application Server ConfigurationOur example application server configuration is below. The important part you need to modify for your own purposes is contained at the bottom of the file and is annotated with comments. The top section tells syslog-ng to route the various system log messages to the appropriate files (much like the system syslog daemon would do). The bottom section then defines a rails-* facility and tells syslog-ng to send those logs to the IP address you specify.
options {
chain_hostnames(off);
long_hostnames (on);
use_fqdn (on);
sync (0);
stats(43200);
log_msg_size(1048576);
};
source s_sys {
file ("/proc/kmsg" log_prefix("kernel: "));
unix-stream ("/dev/log");
internal();
};
destination d_cons { file("/dev/console"); };
destination d_mesg { file("/var/log/messages"); };
destination d_auth { file("/var/log/secure"); };
destination d_mail { file("/var/log/maillog" sync(10)); };
destination d_spol { file("/var/log/spooler"); };
destination d_boot { file("/var/log/boot.log"); };
destination d_cron { file("/var/log/cron"); };
destination d_kern { file("/var/log/kern"); };
destination d_mlal { usertty("*"); };
filter f_kernel { facility(kern); };
filter f_default { level(info..emerg) and
not (facility(mail)
or facility(authpriv)
or facility(cron)); };
filter f_auth { facility(authpriv); };
filter f_mail { facility(mail); };
filter f_emergency { level(emerg); };
filter f_news { facility(uucp) or
(facility(news)
and level(crit..emerg)); };
filter f_boot { facility(local7); };
filter f_cron { facility(cron); };
log { source(s_sys); filter(f_kernel); destination(d_kern); };
log { source(s_sys); filter(f_default); destination(d_mesg); };
log { source(s_sys); filter(f_auth); destination(d_auth); };
log { source(s_sys); filter(f_mail); destination(d_mail); };
log { source(s_sys); filter(f_emergency); destination(d_mlal); };
log { source(s_sys); filter(f_news); destination(d_spol); };
log { source(s_sys); filter(f_boot); destination(d_boot); };
log { source(s_sys); filter(f_cron); destination(d_cron); };
# BBG - Configure the rails portion of our syslog-ng setup!
# This section can be left as is.
log { source(s_sys);
filter(f_rails_apps);
destination(log_server);
flags(final); };
# You should create a filter here that will catch your facility
# you will define within your rails application.
filter f_rails_apps { program("rails-*"); };
# You should set your IP address / port of the remote logging server.
destination log_server {
udp("10.1.1.2" port(515));
};
The logging server is the machine that will collect the log files from the network and combine them into a unified logging server. We define four custom items within this configuration: a source, a destination, a filter, and a log entry. The source tells syslog-ng to listen on an accessible IP (this should either be an internal IP that’s non routable, or should be protected with firewall rules). The destination tells syslog-ng where it should write the combined log files to. The filter is what we use to parse out our rails specific data, and the log line takes those above elements and combines them so syslog-ng knows what to do.
options {
chain_hostnames(off);
sync (0);
dir_owner(root);
dir_group(logs);
stats(43200);
use_dns(yes);
dns_cache(yes);
dns_cache_size(100);
dns_cache_expire(3600);
dns_cache_expire_failed(600);
keep_hostname(yes);
long_hostnames(on);
use_fqdn(no);
log_msg_size(1048576);
log_fifo_size (1000);
};
source s_sys {
file ("/proc/kmsg" log_prefix("kernel: "));
unix-stream ("/dev/log");
internal();
};
destination d_cons { file("/dev/console"); };
destination d_mesg { file("/var/log/messages"); };
destination d_auth { file("/var/log/secure"); };
destination d_mail { file("/var/log/maillog" sync(10)); };
destination d_spol { file("/var/log/spooler"); };
destination d_boot { file("/var/log/boot.log"); };
destination d_cron { file("/var/log/cron"); };
destination d_kern { file("/var/log/kern"); };
destination d_mlal { usertty("***"); };
filter f_kernel { facility(kern); };
filter f_default { level(info..emerg) and
not (facility(mail)
or facility(authpriv)
or facility(cron)); };
filter f_auth { facility(authpriv); };
filter f_mail { facility(mail); };
filter f_emergency { level(emerg); };
filter f_news { facility(uucp) or
(facility(news)
and level(crit..emerg)); };
filter f_boot { facility(local7); };
filter f_cron { facility(cron); };
log { source(s_sys); filter(f_kernel); destination(d_kern); };
log { source(s_sys); filter(f_default); destination(d_mesg); };
log { source(s_sys); filter(f_auth); destination(d_auth); };
log { source(s_sys); filter(f_mail); destination(d_mail); };
log { source(s_sys); filter(f_emergency); destination(d_mlal); };
log { source(s_sys); filter(f_news); destination(d_spol); };
log { source(s_sys); filter(f_boot); destination(d_boot); };
log { source(s_sys); filter(f_cron); destination(d_cron); };
# BBG - Define a source for log files. This causes syslog-ng to
# listen on an IP that then allows your application server
# syslog-ng servers to talk to.
source app_cluster {
udp( ip(10.1.1.2));
};
# BBG - Define a destination. Our filter will then map anything
# from our above source to this destination.
destination d_rails_apps {
file("/var/log/rails/prod/$PROGRAM/$YEAR-$MONTH/$DAY/$PROGRAM-$YEAR$MONTH$DAY"
owner("deploy")
group("logs")
perm(0660)
dir_perm(0770)
create_dirs(yes)
);
};
# BBG - Define our filter that we'll use to map our logs.
filter f_rails_apps { program("rails-***"); };
# BBG - Now use the above source and destination to route our
# log file to the appropriate place.
log { source(app_cluster);
filter (f_rails_apps);
destination(d_rails_apps);
flags(final); };
On each one of your servers, start syslog-ng. It should start cleanly:
[root@log rails]# /etc/init.d/syslog-ng start Starting syslog-ng: [ OK ]
If you get errors, it probably means you typo’d something in your configuration. Take a look at the output, fix the error, and try restarting syslog-ng.
Step 4: Install the gemOn your application servers, install the syslog_logger gem:
[root@staging-app01 environments]# gem install SyslogLogger Successfully installed SyslogLogger-1.4.0 1 gem installed Installing ri documentation for SyslogLogger-1.4.0... Installing RDoc documentation for SyslogLogger-1.4.0...
Within your application, you’ll want to add the following to your config/environments/production.rb (or config/environments/staging.rb). You can pass an argument to SyslogLogger.new to define the “facility” to use if you want to separate staging logs from production logs. Make sure to start your facility with rails- so our filters above work.
It should be noted that our syslog-ng configuration will separate apps based on this token, so if you’re running a staging and production app, you’d want to name them uniquely: rails-my_app-staging / rails-my_app-prod.
# Use a different logger for distributed setups
config.gem 'SyslogLogger', :lib => 'syslog_logger'
require 'syslog_logger'
config.logger = SyslogLogger.new ('rails-my_app-staging')
With a simple application restart, you should now have unified logs! Take a peak in /var/log/rails on your logging server and you should have as much data as you care to digest.
That’s it! You now should have a unified production log. This should simplify debugging for production errors and generally make your life much happier.
Additionally, you can now start to play with cool features such as the Production Log Analyzer
Blue Box Group, LLC
The Latest
Archives
- February 2012
- January 2012
- December 2011
- August 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- March 2010
- February 2010
- January 2010
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- April 2009
- March 2009
- February 2009
- January 2009
- September 2008
- August 2008
- July 2008
- June 2008
- May 2008
- January 2008
- December 2007
- October 2007
- August 2007
