LogAgent 2.1, log file recollection tool By Floydman, floydian_99@yahoo.com August 11th, 2002 This paper is available online at www.geocities.com/floydian_99 and http://securit.iquebec.com This paper can be freely distributed and reproduced, as long as correct credentials are maintained, and that no modifications are made to this file. For corrections, suggestions or comments, please send me an e-mail. Abstract The goal of this paper is to present LogAgent 2.1, a tool made in Perl for recollecting log files from various applications and various machines into a central location in (almost) real-time, in order to improve the administrator's network activity awareness. Preface It has been mentionned many time by me and my others that centralization of log files is crucial is network administration is we take security seriously. These log files could be produced by antivirus engines, personal firewalls, download managers, or even the command prompt history (using ComLog). When comes the time to choose computer security tools, one of the most important feature should be the ability to centralize the information contained in the log files. This allows for quicker understanding and better response from the admins, ans it prevents the evidence from being tampered by a potential intruder. So because of this, somehow good products could be overlooked simply because they fail to provide this single feature, and sometimes this leads to purchasing a product that offers (and sells) many other features not necessarily needed, or products that are not as flexible as desired when comes the time to make it work on your environment. In order to resolve this, I programmed LogAgent, now at version 2.1, which is an agent that you can run on all your Windows machines to monitor the log files of various unrelated applications and to redirect any new input made to these files to one or many central locations (a shared directory on one of your servers or admin station). Targeted audience This document is presented to anyone who has interests in computer security, NT/2K Administration, computer monitoring, intrusion detection, Perl programming and computing in general. Table of contents 1. What is LogAgent? 2. History behind LogAgent 3. Version History 4. Known issues 5. To install 6. Source code 7. Sample config.txt 8. sample mondir.txt 1. What is LogAgent? LogAgent is a piece of software made in Perl designed to monitor ascii log files and redirect any change made to it to a central location. The purpose of this is to add flexibility in some security (or other) applications on the choice of destination folder for the log files. The ability to specify your own destination folder for log files could be a crucial requirement in your specification for a security software, and good products can be overlooked simply because they lack this single feature. LogAgent tries to fill that gap by monitoring the log files on the local machine, and then redirects any new line appended to it to the destination of your choice, either on another folder on the same machine or to a remote server for network-wide log file centralization. 2. History behind LogAgent First of all, I'd like to thank Amine Moulay Ramdane, who programmed the AdvNotify PPM package. AdvNotify provides the functions needed to monitor directories and report when changes are made to these directories. LogAgent 1.0 was created in september 2000. At the time, I had written a paper about antivirus protection and deployment in an enterprise network, and one of the topics covered in this paper was the benefits from the centralization of the antivirus logfiles on the network. As I played with other antivirus products and some personnal firewalls, I found out that some of them had their log file destination path hardcoded in the software, making it impossible to choose a different location, and thus preventing the centralization of log files scattered around on a network. The original paper of LogAgent 1.0 was presented at the Seguridad en Computo conference in Mexico City in November 2000. Back then, it was a simple script that I now consider to be "proof of concept" code, but not quite ready to handle the job in the real world. Version 2.0 beta was released in April 2002, which was a big improvement over the previous version. It allowed for more flexible configuration and was cleaner and more structured code. Version 2.1 fixed the design flaw that caused log lines to be dropped when changes were quicker than what LogAgent could keep track of. 3. Version History Changes from version 2.0 beta include: - you can specify filenames instead of simply its path, which allows for keeping track of linecount. If you only wnat to specify a logfile path without the actual filename (which is how LogAgent used to work), simply put a dummy filename with NO extensions - keeping track of linecount allows to fix the line-drop problem previous versions encountered. If the filename is omitted, linecount is still taken care of. If lines are deleted from a file when it is being monitored, LogAgent will reset the linecount to 0 and redump the whole file. Changes from version 1.0 include: - the configuration is now independant of source-code (big improvement). The configuration is now handled by two files, config.txt and mondir.txt - the programming style is procedural, so it makes it for simpler code to read - Lots of comments in the code to make it easier to understand how the program works - checks the config files for empty spaces, empty lines, bad characters, switch '/' to '\', ... - can have as many output directories (local or remote) as you want (one directory path per line in mondir.txt) - can have as many monitored directories (local) as you want, listed one directory path by line after the fourth line in config.txt - the ability to switch on and off the logging of the IP, hostname and username (first 3 lines of config.txt) - the ability to switch on and off the display of information on the console (the fouthr line of config.txt) - using the SHOWCONSOLE option, you can actively keep an eye on the activity in your log files by monitoring your central log directory with LogAgent 4. Known issues There are some known issues with this program. Nothing too bad, but it is good for you to know about these little quirks in case you find that LogAgent is not behaving the way you expected. - in mondir.txt, make sure that each entry is a full path and file name. For log files that you don't know in advance the name of the file, put a fake filename with no extensions (such as 'dummy'). - in config.txt, make sure that all your path names end with a '\' - Not really an issue, but more of an installation note, if you plan to install LogAgent manually (without the install pack). You need to install the AdvNotify and Win32 API Perl modules available at http://www.generation.net/~aminer/Perl/. However, this site now appears to be down, so you can find the perl modules at http://www.geocities.com/floydian_99/Win32-AdvNotify.tar.gz and http://www.geocities.com/floydian_99/Win32-API-0.20.zip. If you use LogAgent on a single machine, this will work fine, but if you want to deploy it on a network you need to change the DLL_PATH variable in AdvNotify.pm from "$DLLPath =$Config{installsitearch}."\\auto\\Win32\\AdvNotify\\advnotify.dll";" to "$DLLPath =$ENV{SystemRoot}."\\System32\\advnotify.dll";" before compilation, and copy the file advnotify.dll to \winnt\system32. Advnotify.pm and AdvNotify.dll are buried deep in the Perl tree, so just make a file find to locate them. This way it will be easier to deploy. - On NT4, when you do a File, Find, there may be a delay before the Find window appears. This is caused somehow by AdvNotify, but don't know more about it. Other issues have currently been fixed. 5. To install These have been tested on WinNT4 SP3 and above. It should normally work on Win 2K, but this have not been tested properly yet. To install LogAgent, you have two options: 1) Download the freeware version installation package at http://securit.iquebec.com/download.html (or purchase the full version); or 2) Installing it manually from the source code displayed below. In the case of 1), simply download the file and execute it on the machine you want to install it on. This file will copy 4 files in your \Winnt directory (logagent21.exe, logagent21.lnk, mondir.txt, config.txt), plus 1 file in \winnt\system32 (advnotify.dll). It will also add a key in HKLocalMachine\Software\Microsoft\Windows\CurrentVersion\Run to enable LogAgent at each startup. Before starting LogAgent, you have to configure it using the files mondir.txt and config.txt in \winnt. If the config files point to a directory that is not existing, it will refuse to work. In the case of 2), first of all you need a copy of Perl installed, I suggest ActivePerl from www.activestate.com (it's free) if you don't already have one. Then, you have to download the Win32API and AdvNotify perl modules and install them. If you can't find these two modules froma Google Search or from the website displayed in the code comments, you can download them at http://www.geocities.com/floydian_99/Win32-AdvNotify.tar.gz and http://www.geocities.com/floydian_99/Win32-API-0.20.zip. Once unzipped, go to the directory containing the files and type 'ppm install win32-api.ppd' and 'ppm install advnotify.ppd'. Once this is done, you may want to change the $DLLPath variable in AdvNotify.pm as discussed in the previous chapter (do not forget to copy advnotify.dll to the \winnt\system32\ directory). Now, copy the source code presented in chapter 6 and save it in a file called logagent.pl. Create the configuration files config.txt and mondir.txt as shown in chapters 7 and 8, and place these files in the same folder as logagent.pl. Then, you only have to execute logagent.pl, or alternatively compile it with perl2exe.exe to produce logagent.exe. To make it start automatically at each boot, you can put a shortcut in the Startup folder, or create a key in HKLocalMachine\Software\Microsoft\Windows\CurrentVersion\Run. 6. Source code logagent21.pl #! C:\perl\bin\perl.exe # LogAgent 2.1 ######################################################################################### # LogAgent 2.1 # # by Floydman floydian_99@yahoo.com # # Copyright 2002 SecurIT Informatique Inc. http://securit.iquebec.com # # # # This program gets its configuration from the file config.txt, and the list of files or# # directories to be monitored from the file mondir.txt. These two files have to be in # # the same directory as LogAgent. The config file lets you specify if you want to # # include the IP of the machine, the hostname and the username in the log files, in # # these cases where the software generating the log doesn't provide these credentials. # # You can also specify to display entries captured by LogAgent on the console or not. # # Then, the program starts the monitoring threads for each directory entry in # # mondir.txt, and then enters in an infinite-loop, waiting for signals from the moni- # # toring threads. When a signal is trigerred (ie: a file as changed in the directory # # you are monitoring), it gets the appended lines from the log file, and sends it to # # the specified outputs. Output dirs can be remote or local, and as many as you want. # # # # note about config.txt: Do not modify the headers LOGIP, LOGHOST, LOGUSER and # # SHOWCONSOLE, or the program will stop working. Only change the Y or N at the end of # # the line. Also make sure that pathnames ends with \ # # # # note about mondir.txt: If you know the exact name of the log file you wish to monitor # # then you should type its whole path, for example c:\Antivirus\virlog.txt. # # If you only know the path of the log file, but cannot determine in advance it exact # # name (if the name is random-generated, or date-generated), then put the path name # # followed by a dummy filename WITH NO EXTENSION, for exemple C:\WINNT\Logs\dummy # ######################################################################################### ######################################################################################### # LICENSE # # This software is Open Source. This means that its source code is open, free and avai-# # lable for anyone to look into, make modifications, correct bugs (let me know, please) # # and use for their personal use. You can create your own binaries with the evaluation # # version of perl2exe (www.indigostar.com), or download the shareware version install # # package available on my website. If you own perl2exe Pro, you are allowed to compile # # LogAgent and distribute it within your organization only. DO NOT distribute such a # # compiled version to third parties. The full version of LogAgent is to be distributed # # by SecurIT Informatique Inc or its distribution partners only. Send an e-mail to # # securit@iquebec.com if you would like to be a LogAgent distributor. # # LogAgent (full version) is available for commercial use at 15$US per copy, volume # # discounts are available starting at 100 copies, and any purchase over 10 copies # # comes with 1 year of free e-mail support. # # LogAgent (full version) is compiled with perl2exe Pro, which means that no banner is # # displayed at the end of execution. Also, it can run as a background process, making # # it invisible to desktop users. The codebaseis the same. # ######################################################################################### ######################################################################################### # Main Program # # This is the main structure of LogAgent. # # This procedure takes note of the machine credentials, LogAgent's configuration and # # the list of directories and files to monitor. It also gets the linecount of the # # files to monitor. # # Then, we start a thread for each entry in mondir.txt and we enter in the main loop. # # This loop waits for signals from the threads, and when a signal is received, it # # captures the last lines of the modified log file. These linee are then sent to the # # various outputs specified in config.txt. # # At the end of the loop (CTRL-C) we destroy our threads and memory objects, for clean # # programming purposes. # ######################################################################################### # Using Win32::AdvNotify # By Amine Moulay Ramdane # Website: http://www.generation.net/~aminer/Perl/ # This Perl module is the core engine of LogAgent. This module contains all the funtionalities # For monitoring the changes made to files and folders on the system # You will also need to install the Win32 API Perl module in order to use AdvNotify use Win32::AdvNotify qw(FILE_NAME SIZE INFINITE Yes No All %ActionName %ActionColor); # Declaration of needed components for machine identification use Socket; use Sys::Hostname; use File::Basename; my $element; # Creation of the AdvNotify object my $obj = new Win32::AdvNotify()|| die "Can't create object\n"; # Filelocking of the config files lockconfig(); # Creation of machine ID table @id = getid(); # Creation of config table my @config = getconfig(); # Creation of file table my @filelist = getfilelist(); # Creation of line count table my @linecount = getlinecount (@filelist); # Creation of mondir table my @mondir = getmondir(); # Creation of threads table. Threads are started, and then launched, this is the way the AdvNotify module works my $index=0; foreach $element (@mondir) { $threads[$index] = $obj->StartThread(Directory => $mondir[$index], Filter => All , WatchSubtree => No ) || die "Can't start thread\n"; $threads[$index]->EnableWatch() || die "Problem starting EnableWatch()\n"; $index++; } print "Log Agent 2.1, brought to you by Floydman\n"; print "Copyright 2002 SecurIT Informatique Inc.\n"; print "http://securit.iquebec.com\n"; print "\nPress CTRL-C to quit\n"; # Enters the main monitoring loop startmonitoringloop(); # Unlocking of the config files unlockconfig(); # termination of the threads. for ($a; $a<$index; $a++) { $threads[$a]->Terminate(); } # destruction of the object undef $obj; # End of program# ######################################################################################### # procedure lockconfig() # # This procedure locks the config files to prevent tampering while LogAgent runs. # ######################################################################################### sub lockconfig { open(CONFIGFILE," || die "Can't read logip from config.txt"; ($logip=~m/LOGIP/i) || die "LOGIP entry missing in config.txt"; $loghost = || die "Can't read loghost from config.txt"; ($loghost=~m/LOGHOST/i) || die "LOGHOST entry missing in config.txt"; $loguser = || die "Can't read loguser from config.txt"; ($loguser=~m/LOGUSER/i) || die "LOGUSER entry missing in config.txt"; $showconsole = || die "Can't showconsole read from config.txt"; ($showconsole=~m/SHOWCONSOLE/i) || die "SHOWCONSOLE entry missing in config.txt"; while (defined($dir = )) { $dirtable[$j]=$dir; $j++; } ($j==0) && die "No destination directory specifed in config.txt."; @configtable = ($logip, $loghost, $loguser, $showconsole, @dirtable); @configtable = parse(@configtable); (($numarg=@configtable)<5) && die "Not enough parameters in config.txt. Check file for errors."; # Tranformation of the first 4 lines of configtable to boolean value $configtable[0]=$configtable[0]=~m/Y/i; $configtable[1]=$configtable[1]=~m/Y/i; $configtable[2]=$configtable[2]=~m/Y/i; $configtable[3]=$configtable[3]=~m/Y/i; return (@configtable); } ######################################################################################### # procedure getfilelist() # # This procedure gets the list of files to check for in mondir.txt. # ######################################################################################### sub getfilelist { my @filetable; my $k = 0; @temptable; while (defined($file = )) { $temptable[$k]=$file; $k++; } ($k==0) && die "No file to monitor in mondir.txt."; @temptable = parse(@temptable); $k=0; foreach $file (@temptable) { if ($file=~m/.*\w+\.\w{0,3}/) {$filetable[$k]=$file; $k++; } } return (@filetable); } ######################################################################################### # procedure getlinecount (@filelist) # # This procedure gets the list of directories to watch in mondir.txt. # ######################################################################################### sub getlinecount { my (@table) = @_; my @linecount; my @temp; $z = 0; foreach $file (@table) { open (FILE, $file) || die "Can't open ".$file." for line count."; @temp = ; close (FILE); @temp = parse(@temp); $linecount[$z] = @temp; $z++; } return (@linecount); } ######################################################################################### # procedure getmondir() # # This procedure gets the list of directories to watch in mondir.txt. # ######################################################################################### sub getmondir { my @dirtable =(), temptable; my $w = 0; sysseek MONDIRFILE,0,0; while (defined($dir = )) { $temptable[$w]=$dir; $w++; } ($w==0) && die "No directory to monitor in mondir.txt."; @temptable = parse(@temptable); $w=0; foreach $entry (@temptable) { $entry = dirname($entry)."/"; TAG: for ($t=0; $t <= $w; $t++) { if ($entry eq $dirtable[$t]) { last TAG;} if ($t >= $w) { $dirtable[$w] = $entry; $w++; last TAG;} } } return (@dirtable); } ######################################################################################### # procedure parse(table_file) # # This procedure cleans the files from non-valid and blank characters that could be # # placed in the config files. The procedure returns the file as a table. # ######################################################################################### sub parse { my (@table) = @_; #check for invalid characters in table_file chomp @table; foreach $element (@table) { $element=~s%^\s+%%; @char = split (//, $element); foreach $char (@char) { $char=~s%\\%/%; } $element = join ('',@char); } my @tabletemp; my $x = 0; foreach $element (@table) { if ($element ne '') { $tabletemp[$x]=$element; $x++;} } @table = @tabletemp; return (@table); } ######################################################################################### # procedure startmonitoringloop() # # This procedure is the main monitoring loop. When a change is detected in a file # # located in a monitored directory (preferably ASCII files), the procedure calls # # getlastline() with the name of the modified file. The captured line is then sent # # via the procedure sendoutput(), along with LogAgent's configuration table. # ######################################################################################### sub startmonitoringloop { while($threads[0]->Wait(INFINITE))# exit with [Ctrl-C] signal { while($threads[0]->Read(\@data))# exit when the list is empty { for($i=0;$i<=$#data;$i++) { getlastline($data[$i]); } } } } ######################################################################################### # procedure getlastline(filename) # # This procedure gets the last line (non-blank) of the file received as the argument. # # It returns the filename (whitout path) and the last line of the file. # ######################################################################################### sub getlastline {my ($data) = @_; $y = 0; open (LOGFILE, $data->{Directory}.$data->{FileName}) or die "Can't open log file ".$data->{Directory}.$data->{FileName}; flock (LOGFILE, 1) or die "Can't lock file"; @lines = ; close (LOGFILE) or die "Can't close file"; # To unlock the file as fast as possible for new entries @lines = parse(@lines); COUNTER: foreach $file (@filelist) { if ($file ne $data->{Directory}.$data->{FileName}) {$y++;} if ($file eq $data->{Directory}.$data->{FileName}) {last COUNTER;} } if ($filelist[$y] eq '') {$filelist[$y] = $data->{Directory}.$data->{FileName}} if ($linecount[$y] > @lines) { $linecount[$y]=0; } for ($toto=$linecount[$y]; $toto < @lines; $toto++ ) { $lastline = $lines[$linecount[$y]]; $linecount[$y]++; sendoutput($data->{FileName}, $lastline,@config); } } ######################################################################################### # procedure sendoutput(line, config) # # This procedure receives as arguments: the name of the modified file, the last line # # of the logfile, and then the config table (LOGIP, LOGHOST, LOGUSER, SHOWCONSOLE, and # # the various destination directories). The procedure checks the configuration to see # # if it has to append any information to the original line or not. If SHOWCONSOLE in # # enabled, then the line is printed on the screen, if not it simply passes to the next # # step which is to forward this line to all mentionned destinations in config.txt. # ######################################################################################### sub sendoutput { my ($filename, $line, $logip, $loghost, $loguser, $showconsole, @dest) = @_; my $newline=""; if ($logip) {$newline=$newline.$id[0]." ";} if ($loghost) {$newline=$newline.$id[1]." ";} if ($loguser) {$newline=$newline.$id[2]." ";} $newline=$newline.$line; if ($showconsole) {print "$newline\n";} foreach $destdir (@dest) { $destination=$destdir.$filename; open (DEST, ">>".$destination) || die "Can't open master log file $destination"; flock (DEST, 2) || die "Can't lock file for writing"; print DEST "$newline\n" || die "Can't write to file"; close (DEST) || die "Can't close master log file"; } } #EOF 7. Sample config.txt LOGIP=Y LOGHOST=N LOGUSER=Y SHOWCONSOLE=N D:\log\ \\logserver1\shared_floder\ \\logserver2\hidden_share$\ 8. sample mondir.txt D:\Winnt\Internet Logs\ZALog.txt D:\Program Files\Antivirus Software\Log\antivirus.log C:\Winnt\Help\Tutor\dummy