LogAgent 2.1, log file recollection tool
By Floydman, floydian_99@yahoo.com
August 11th, 2002

This paper is available online at www.geocities.com/floydian_99 and 
http://securit.iquebec.com

This paper can be freely distributed and reproduced, as long as correct 
credentials are maintained, and that no modifications are made to this 
file.  For corrections, suggestions or comments, please send me an e-mail.

Abstract

The goal of this paper is to present LogAgent 2.1, a tool made in Perl for 
recollecting log files from various applications and various machines into 
a central location in (almost) real-time, in order to improve the 
administrator's network activity awareness.

Preface

It has been mentionned many time by me and my others that centralization of 
log files is crucial is network administration is we take security 
seriously.  These log files could be produced by antivirus engines, 
personal firewalls, download managers, or even the command prompt history 
(using ComLog).  When comes the time to choose computer security tools, one 
of the most important feature should be the ability to centralize the 
information contained in the log files.  This allows for quicker 
understanding and better response from the admins, ans it prevents the 
evidence from being tampered by a potential intruder.  So because of this, 
somehow good products could be overlooked simply because they fail to 
provide this single feature, and sometimes this leads to purchasing a 
product that offers (and sells) many other features not necessarily needed, 
or products that are not as flexible as desired when comes the time to make 
it work on your environment.  In order to resolve this, I programmed 
LogAgent, now at version 2.1, which is an agent that you can run on all 
your Windows machines to monitor the log files of various unrelated 
applications and to redirect any new input made to these files to one or 
many central locations (a shared directory on one of your servers or admin 
station).

Targeted audience

This document is presented to anyone who has interests in computer 
security, NT/2K Administration, computer monitoring, intrusion detection, 
Perl programming and computing in general.

Table of contents

1. What is LogAgent?
2. History behind LogAgent
3. Version History
4. Known issues
5. To install
6. Source code
7. Sample config.txt
8. sample mondir.txt


1. What is LogAgent?

LogAgent is a piece of software made in Perl designed to monitor ascii log 
files and redirect any change made to it to a central location.  The 
purpose of this is to add flexibility in some security (or other) 
applications on the choice of destination folder for the log files.  The 
ability to specify your own destination folder for log files could be a 
crucial requirement in your specification for a security software, and good 
products can be overlooked simply because they lack this single 
feature.  LogAgent tries to fill that gap by monitoring the log files on 
the local machine, and then redirects any new line appended to it to the 
destination of your choice, either on another folder on the same machine or 
to a remote server for network-wide log file centralization.

2. History behind LogAgent

First of all, I'd like to thank Amine Moulay Ramdane, who programmed the 
AdvNotify PPM package.  AdvNotify provides the functions needed to monitor 
directories and report when changes are made to these 
directories.  LogAgent 1.0 was created in september 2000.  At the time, I 
had written a paper about antivirus protection and deployment in an 
enterprise network, and one of the topics covered in this paper was the 
benefits from the centralization of the antivirus logfiles on the 
network.  As I played with other antivirus products and some personnal 
firewalls, I found out that some of them had their log file destination 
path hardcoded in the software, making it impossible to choose a different 
location, and thus preventing the centralization of log files scattered 
around on a network.

The original paper of LogAgent 1.0 was presented at the Seguridad en 
Computo conference in Mexico City in November 2000.  Back then, it was a 
simple script that I now consider to be "proof of concept" code, but not 
quite ready to handle the job in the real world.  Version 2.0 beta was 
released in April 2002, which was a big improvement over the previous 
version.  It allowed for more flexible configuration and was cleaner and 
more structured code.  Version 2.1 fixed the design flaw that caused log 
lines to be dropped when changes were quicker than what LogAgent could keep 
track of.

3. Version History

Changes from version 2.0 beta include:

- you can specify filenames instead of simply its path, which allows for 
keeping track of linecount.  If you only wnat to specify a logfile path 
without the actual filename (which is how LogAgent used to work), simply 
put a dummy filename with NO extensions

- keeping track of linecount allows to fix the line-drop problem previous 
versions encountered.  If the filename is omitted, linecount is still taken 
care of.  If lines are deleted from a file when it is being monitored, 
LogAgent will reset the linecount to 0 and redump the whole file.

Changes from version 1.0 include:

- the configuration is now independant of source-code (big 
improvement).  The configuration is now handled by two files, config.txt 
and mondir.txt

- the programming style is procedural, so it makes it for simpler code to read

- Lots of comments in the code to make it easier to understand how the 
program works

- checks the config files for empty spaces, empty lines, bad characters, 
switch '/' to '\', ...

- can have as many output directories (local or remote) as you want (one 
directory path per line in mondir.txt)

- can have as many monitored directories (local) as you want, listed one 
directory path by line after the fourth line in config.txt

- the ability to switch on and off the logging of the IP, hostname and 
username (first 3 lines of config.txt)

- the ability to switch on and off the display of information on the 
console (the fouthr line of config.txt)

- using the SHOWCONSOLE option, you can actively keep an eye on the 
activity in your log files by monitoring your central log directory with 
LogAgent

4. Known issues

There are some known issues with this program.  Nothing too bad, but it is 
good for you to know about these little quirks in case you find that 
LogAgent is not behaving the way you expected.

- in mondir.txt, make sure that each entry is a full path and file 
name.  For log files that you don't know in advance the name of the file, 
put a fake filename with no extensions (such as 'dummy').

- in config.txt, make sure that all your path names end with a '\'

- Not really an issue, but more of an installation note, if you plan to 
install LogAgent manually (without the install pack).  You need to install 
the AdvNotify and Win32 API Perl modules available at 
http://www.generation.net/~aminer/Perl/.  However, this site now appears to 
be down, so you can find the perl modules at 
http://www.geocities.com/floydian_99/Win32-AdvNotify.tar.gz and 
http://www.geocities.com/floydian_99/Win32-API-0.20.zip.  If you use 
LogAgent on a single machine, this will work fine, but if you want to 
deploy it on a network you need to change the DLL_PATH variable in 
AdvNotify.pm from "$DLLPath 
=$Config{installsitearch}."\\auto\\Win32\\AdvNotify\\advnotify.dll";" to 
"$DLLPath =$ENV{SystemRoot}."\\System32\\advnotify.dll";" before 
compilation, and copy the file advnotify.dll to 
\winnt\system32.  Advnotify.pm and AdvNotify.dll are buried deep in the 
Perl tree, so just make a file find to locate them.  This way it will be 
easier to deploy.

- On NT4, when you do a File, Find, there may be a delay before the Find 
window appears.  This is caused somehow by AdvNotify, but don't know more 
about it.

Other issues have currently been fixed.

5. To install

These have been tested on WinNT4 SP3 and above.  It should normally work on 
Win 2K, but this have not been tested properly yet.

To install LogAgent, you have two options:

1) Download the freeware version installation package at 
http://securit.iquebec.com/download.html (or purchase the full version);

or 2) Installing it manually from the source code displayed below.

In the case of 1), simply download the file and execute it on the machine 
you want to install it on.  This file will copy 4 files in your \Winnt 
directory (logagent21.exe, logagent21.lnk, mondir.txt, config.txt), plus 1 
file in \winnt\system32 (advnotify.dll).  It will also add a key in 
HKLocalMachine\Software\Microsoft\Windows\CurrentVersion\Run to enable 
LogAgent at each startup.  Before starting LogAgent, you have to configure 
it using the files mondir.txt and config.txt in \winnt.  If the config 
files point to a directory that is not existing, it will refuse to work.

In the case of 2), first of all you need a copy of Perl installed, I 
suggest ActivePerl from www.activestate.com (it's free) if you don't 
already have one.  Then, you have to download the Win32API and AdvNotify 
perl modules and install them.  If you can't find these two modules froma 
Google Search or from the website displayed in the code comments, you can 
download them at 
http://www.geocities.com/floydian_99/Win32-AdvNotify.tar.gz and 
http://www.geocities.com/floydian_99/Win32-API-0.20.zip.  Once unzipped, go 
to the directory containing the files and type 'ppm install win32-api.ppd' 
and 'ppm install advnotify.ppd'.  Once this is done, you may want to change 
the $DLLPath variable in AdvNotify.pm as discussed in the previous chapter 
(do not forget to copy advnotify.dll to the \winnt\system32\ directory).

Now, copy the source code presented in chapter 6 and save it in a file 
called logagent.pl.  Create the configuration files config.txt and 
mondir.txt as shown in chapters 7 and 8, and place these files in the same 
folder as logagent.pl.  Then, you only have to execute logagent.pl, or 
alternatively compile it with perl2exe.exe to produce logagent.exe.  To 
make it start automatically at each boot, you can put a shortcut in the 
Startup folder, or create a key in 
HKLocalMachine\Software\Microsoft\Windows\CurrentVersion\Run.

6. Source code

logagent21.pl
#! C:\perl\bin\perl.exe
# LogAgent 2.1

#########################################################################################
# LogAgent 2.1                    #
# by Floydman  floydian_99@yahoo.com              #
# Copyright 2002 SecurIT Informatique Inc.  http://securit.iquebec.com      #
#                      #
# This program gets its configuration from the file config.txt, and the 
list of files or#
# directories to be monitored from the file mondir.txt.  These two files 
have to be in  #
# the same directory as LogAgent.  The config file lets you specify if you 
want to   #
# include the IP of the machine, the hostname and the username in the log 
files, in  #
# these cases where the software generating the log doesn't provide these 
credentials.  #
# You can also specify to display entries captured by LogAgent on the 
console or not.   #
# Then, the program starts the monitoring threads for each directory entry 
in     #
# mondir.txt, and then enters in an infinite-loop, waiting for signals from 
the moni-  #
# toring threads.  When a signal is trigerred (ie: a file as changed in the 
directory  #
# you are monitoring), it gets the appended lines from the log file, and 
sends it to    #
# the specified outputs.  Output dirs can be remote or local, and as many 
as you want.  #
#                      #
# note about config.txt: Do not modify the headers LOGIP, LOGHOST, LOGUSER 
and     #
# SHOWCONSOLE, or the program will stop working.  Only change the Y or N at 
the end of  #
# the line.  Also make sure that pathnames ends with \          #
#                      #
# note about mondir.txt: If you know the exact name of the log file you 
wish to monitor #
# then you should type its whole path, for example c:\Antivirus\virlog.txt.    #
# If you only know the path of the log file, but cannot determine in 
advance it exact   #
# name (if the name is random-generated, or date-generated), then put the 
path name  #
# followed by a dummy filename WITH NO EXTENSION, for exemple 
C:\WINNT\Logs\dummy  #
#########################################################################################

#########################################################################################
# LICENSE                    #
# This software is Open Source.  This means that its source code is open, 
free and avai-#
# lable for anyone to look into, make modifications, correct bugs (let me 
know, please) #
# and use for their personal use.  You can create your own binaries with 
the evaluation #
# version of perl2exe (www.indigostar.com), or download the shareware 
version install    #
# package available on my website.  If you own perl2exe Pro, you are 
allowed to compile #
# LogAgent and distribute it within your organization only.  DO NOT 
distribute such a   #
# compiled version to third parties.  The full version of LogAgent is to be 
distributed  #
# by SecurIT Informatique Inc or its distribution partners only.  Send an 
e-mail to   #
# securit@iquebec.com if you would like to be a LogAgent distributor.      #
# LogAgent (full version) is available for commercial use at 15$US per 
copy, volume  #
# discounts are available starting at 100 copies, and any purchase over 10 
copies  #
# comes with 1 year of free e-mail support.            #
# LogAgent (full version) is compiled with perl2exe Pro, which means that 
no banner is  #
# displayed at the end of execution.  Also, it can run as a background 
process, making  #
# it invisible to desktop users.  The codebaseis the same.        #
#########################################################################################

#########################################################################################
# Main Program                    #
# This is the main structure of LogAgent.            #
# This procedure takes note of the machine credentials,  LogAgent's 
configuration and  #
# the list of directories and files to monitor.   It also gets the linecount 
of the   #
# files to monitor.                  #
# Then, we start a thread for each entry in mondir.txt and we enter in the 
main loop.  #
# This loop waits for signals from the threads, and when a signal is 
received, it   #
# captures the last lines of the modified log file.  These linee are then 
sent to the   #
# various outputs specified in config.txt.            #
# At the end of the loop (CTRL-C) we destroy our threads and memory 
objects, for clean   #
# programming purposes.                  #
#########################################################################################

# Using Win32::AdvNotify
# By Amine Moulay Ramdane <aminer@generation.net>
# Website: http://www.generation.net/~aminer/Perl/
# This Perl module is the core engine of LogAgent.  This module contains 
all the funtionalities
# For monitoring the changes made to files and folders on the system
# You will also need to install the Win32 API Perl module in order to use 
AdvNotify

use Win32::AdvNotify qw(FILE_NAME SIZE INFINITE Yes No
                         All %ActionName %ActionColor);

# Declaration of needed components for machine identification
use Socket;
use Sys::Hostname;
use File::Basename;
my $element;

# Creation of the AdvNotify object
my $obj  = new Win32::AdvNotify()|| die "Can't create object\n";

# Filelocking of the config files
lockconfig();

# Creation of machine ID table
@id = getid();

# Creation of config table
my @config = getconfig();

# Creation of file table
my @filelist = getfilelist();

# Creation of line count table
my @linecount = getlinecount (@filelist);

# Creation of mondir table
my @mondir = getmondir();

# Creation of threads table.  Threads are started, and then launched, this 
is the way the AdvNotify module works
my $index=0;
foreach $element (@mondir)
  {
  $threads[$index] = $obj->StartThread(Directory    => $mondir[$index],
                              Filter       =>  All ,
                              WatchSubtree =>  No ) || die "Can't start 
thread\n";
  $threads[$index]->EnableWatch() || die "Problem starting EnableWatch()\n";
  $index++;
  }

print "Log Agent 2.1, brought to you by Floydman\n";
print "Copyright 2002 SecurIT Informatique Inc.\n";
print "http://securit.iquebec.com\n";
print "\nPress CTRL-C to quit\n";

# Enters the main monitoring loop
startmonitoringloop();

# Unlocking of the config files
unlockconfig();

# termination of the threads.
for ($a; $a<$index; $a++)
    { $threads[$a]->Terminate(); }

# destruction of the object
undef $obj;

# End of program#

#########################################################################################
# procedure lockconfig()                #
# This procedure locks the config files to prevent tampering while LogAgent 
runs.  #
#########################################################################################
sub lockconfig
{
open(CONFIGFILE,"<config.txt") || die "Can't open config.txt";
open(MONDIRFILE,"<mondir.txt") || die "Can't open mondir.txt";
}

#########################################################################################
# procedure unlockconfig()                #
# This procedure unlocks the config files.            #
#########################################################################################
# Unlocking of the config files
sub unlockconfig
{
close (CONFIGFILE) || die "Can't close config.txt";
close (MONDIRFILE) || die "Can't close mondir.txt";
}


#########################################################################################
# procedure getid()                   #
# This procedure gets the IP address, the host name and the username of the 
machine.  #
#########################################################################################

sub getid
{
# Define username, IP address and hostname of the local machine
my $addr = inet_ntoa(scalar(gethostbyname($name)) || 'localhost');
my $host = hostname() || "hostname not defined";
my $login = getlogin || getpwuid($<) || "not logged";
my @id_table = ($addr, $host, $login);
return (@id_table);

}

#########################################################################################
# procedure getconfig()                 #
# This procedure gets the configuration file config.txt.        #
#########################################################################################

sub getconfig
{
my @configtable;
my @dirtable;
$j = 0;
$numarg = 0;

$logip = <CONFIGFILE> || die "Can't read  logip from config.txt";
($logip=~m/LOGIP/i) || die "LOGIP entry missing in config.txt";

$loghost = <CONFIGFILE> || die "Can't read loghost from config.txt";
($loghost=~m/LOGHOST/i) || die "LOGHOST entry missing in config.txt";

$loguser = <CONFIGFILE> || die "Can't read loguser from config.txt";
($loguser=~m/LOGUSER/i) || die "LOGUSER entry missing in config.txt";

$showconsole = <CONFIGFILE> || die "Can't showconsole read from config.txt";
($showconsole=~m/SHOWCONSOLE/i) || die "SHOWCONSOLE entry missing in 
config.txt";

while (defined($dir = <CONFIGFILE>))
  {
   $dirtable[$j]=$dir;
   $j++;
  }
($j==0) && die "No destination directory specifed in config.txt.";


@configtable = ($logip, $loghost, $loguser, $showconsole, @dirtable);
@configtable = parse(@configtable);

(($numarg=@configtable)<5) && die "Not enough parameters in 
config.txt.  Check file for errors.";

# Tranformation of the first 4 lines of configtable to boolean value
$configtable[0]=$configtable[0]=~m/Y/i;
$configtable[1]=$configtable[1]=~m/Y/i;
$configtable[2]=$configtable[2]=~m/Y/i;
$configtable[3]=$configtable[3]=~m/Y/i;

return (@configtable);
}

#########################################################################################
# procedure getfilelist()                 #
# This procedure gets the list of files to check for in mondir.txt.       #
#########################################################################################

sub getfilelist
{
my @filetable;
my $k = 0;
@temptable;

while (defined($file = <MONDIRFILE>))
  {
   $temptable[$k]=$file;
   $k++;
  }
($k==0) && die "No file to monitor in mondir.txt.";


@temptable = parse(@temptable);
$k=0;
foreach $file (@temptable)
{ if ($file=~m/.*\w+\.\w{0,3}/) {$filetable[$k]=$file; $k++; } }

return (@filetable);
}

#########################################################################################
# procedure getlinecount (@filelist)              #
# This procedure gets the list of directories to watch in mondir.txt.      #
#########################################################################################
sub getlinecount
{
my (@table) = @_;
my @linecount;
my @temp;
$z = 0;

foreach $file (@table)
   {
    open (FILE, $file) || die "Can't open ".$file." for line count.";
    @temp = <FILE>;
    close (FILE);
    @temp = parse(@temp);
    $linecount[$z] = @temp;
    $z++;
   }

return (@linecount);
}

#########################################################################################
# procedure getmondir()                 #
# This procedure gets the list of directories to watch in mondir.txt.      #
#########################################################################################

sub getmondir
{
my @dirtable =(), temptable;
my $w = 0;

sysseek MONDIRFILE,0,0;
while (defined($dir = <MONDIRFILE>))
  {
   $temptable[$w]=$dir;
   $w++;
  }
($w==0) && die "No directory to monitor in mondir.txt.";


@temptable = parse(@temptable);

$w=0;
foreach $entry (@temptable)
  {
   $entry = dirname($entry)."/";
TAG:  for ($t=0; $t <= $w; $t++)
     {
        if ($entry eq $dirtable[$t]) { last TAG;}
        if ($t >= $w) { $dirtable[$w] = $entry; $w++; last TAG;}
     }
  }
return (@dirtable);
}


#########################################################################################
# procedure parse(table_file)                 #
# This procedure cleans the files from non-valid and blank characters that 
could be  #
# placed in the config files.  The procedure returns the file as a table.    #
#########################################################################################

sub parse
{ my (@table) = @_;

#check for invalid characters in table_file
chomp @table;

foreach $element (@table)
  {
   $element=~s%^\s+%%;
   @char = split (//, $element);

   foreach $char (@char)
     { $char=~s%\\%/%; }
   $element = join ('',@char);
  }

my @tabletemp;
my $x = 0;

foreach $element (@table)
    {
     if ($element ne '') {
  $tabletemp[$x]=$element;
  $x++;}
    }

@table = @tabletemp;
return (@table);
}


#########################################################################################
# procedure startmonitoringloop()              #
# This procedure is the main monitoring loop.  When a change is detected in 
a file  #
# located in a monitored directory (preferably ASCII files), the procedure 
calls   #
# getlastline() with the name of the modified file.  The captured line is 
then sent  #
# via the procedure sendoutput(), along with LogAgent's configuration table.    #
#########################################################################################

sub startmonitoringloop
{

while($threads[0]->Wait(INFINITE))# exit with [Ctrl-C] signal
   {
    while($threads[0]->Read(\@data))# exit when the list is empty
     {
      for($i=0;$i<=$#data;$i++)
       {
        getlastline($data[$i]);
       }
     }
   }
}

#########################################################################################
# procedure getlastline(filename)              #
# This procedure gets the last line (non-blank) of the file received as the 
argument.  #
# It returns the filename (whitout path) and the last line of the file.      #
#########################################################################################

sub getlastline
{my ($data) = @_;
$y = 0;

open (LOGFILE, $data->{Directory}.$data->{FileName}) or die "Can't open log 
file ".$data->{Directory}.$data->{FileName};
flock (LOGFILE, 1) or die "Can't lock file";
@lines = <LOGFILE>;
close (LOGFILE) or die "Can't close file"; # To unlock the file as fast as 
possible for new entries

@lines = parse(@lines);

COUNTER: foreach $file (@filelist)
          {
             if ($file ne $data->{Directory}.$data->{FileName}) {$y++;}
             if ($file eq $data->{Directory}.$data->{FileName}) {last COUNTER;}
          }
if ($filelist[$y] eq '') {$filelist[$y] = $data->{Directory}.$data->{FileName}}
if ($linecount[$y] > @lines) { $linecount[$y]=0; }

for ($toto=$linecount[$y]; $toto < @lines; $toto++ )
  {
   $lastline = $lines[$linecount[$y]];
   $linecount[$y]++;
   sendoutput($data->{FileName}, $lastline,@config);
  }
}

#########################################################################################
# procedure sendoutput(line, config)              #
# This procedure receives as arguments: the name of the modified file, the 
last line  #
# of the logfile, and then the config table (LOGIP, LOGHOST, LOGUSER, 
SHOWCONSOLE, and   #
# the various destination directories).  The procedure checks the 
configuration to see  #
# if it has to append any information to the original line or not.  If 
SHOWCONSOLE in   #
# enabled, then the line is printed on the screen, if not it simply passes 
to the next  #
# step which is to forward this line to all mentionned destinations in 
config.txt.  #
#########################################################################################

sub sendoutput
{ my ($filename, $line, $logip, $loghost, $loguser, $showconsole, @dest) = @_;

my $newline="";

if ($logip) {$newline=$newline.$id[0]." ";}
if ($loghost) {$newline=$newline.$id[1]." ";}
if ($loguser) {$newline=$newline.$id[2]." ";}

$newline=$newline.$line;

if ($showconsole) {print "$newline\n";}

foreach $destdir (@dest)
    {
     $destination=$destdir.$filename;
     open (DEST, ">>".$destination) || die "Can't open master log file 
$destination";
     flock (DEST, 2) || die "Can't lock file for writing";
     print DEST "$newline\n" || die "Can't write to file";
     close (DEST) || die "Can't close master log file";
    }
}

#EOF

7. Sample config.txt
LOGIP=Y
LOGHOST=N
LOGUSER=Y
SHOWCONSOLE=N
D:\log\
\\logserver1\shared_floder\
\\logserver2\hidden_share$\

8. sample mondir.txt
D:\Winnt\Internet Logs\ZALog.txt
D:\Program Files\Antivirus Software\Log\antivirus.log
C:\Winnt\Help\Tutor\dummy