Monitoring application log files is important. If something goes wrong, you want to be notified. This is even more critical for background jobs where no user is facing the error directly.

A state of the art solution for this problem would be ELK, combined with Alerting , but maybe your infrastructure is not at this point yet. For small systems or as a workaround solution, there’s an old Linux tool called logcheck.

Logcheck was actually developed to monitor Linux system log files (e.g., detect a hacker attack), but it can be easily configured for your own application logs as well.

What logcheck can do:

  • every x hours, it would scan a list of log files you specify
  • each line in the logfile is matched against a list of regular expressions you specify
  • all the lines which are matched would be sent out by e-mail

Installation

On Ubuntu, install it like

~$ sudo aptitude install logcheck

Logcheck gets configured in /etc/logcheck. This directory looks like

.
├── cracking.d 
├── cracking.ignore.d 
├── ignore.d.paranoid 
├── ignore.d.server 
├── ignore.d.workstation 
├── violations.d 
├── violations.ignore.d 
├── header.txt
├── logcheck.conf
└── logcheck.logfiles

For our setup, only

.
├── ignore.d.server 
├── violations.d 
├── logcheck.conf
└── logcheck.logfiles

are needed.

Setup

Edit logcheck.conf and enter the e-mail address for your alerts. We keep REPORTLEVEL="server" unchanged, so logcheck will run in server mode. Therefore, we can ignore directories ignore.d.paranoid and ignore.d.workstation completely.

Add all the logfiles you want to monitor to logcheck.logfiles. Placeholders work:

# /etc/logcheck/logcheck.logfiles

/var/log/my_app/*.log

Now, add a new file violations.d/my_app and enter your regex, e.g.:

# /etc/logcheck/violations.d/my_app

^.*error.*$
^.*Error.*$
^.*exception.*$
^.*Exception.*$

Finally, ignore everything else. Add

# /etc/logcheck/ignore.d.server/my_app

^.*$

to ignore.d.server/my_app.

Disclaimer

Be aware of these caveats:

  • it doesn’t scale: it can only act on logs from a single server
  • it can only do regex, cannot detect patterns on multiple rows
  • for low volume alerts only
  • no UI
  • no history (except the e-mail themselves)

However, until you have something like ELK, it might serve you well.