# HG changeset patch # User Matti Hamalainen # Date 1250201998 -10800 # Node ID 26c2cc5077aa3eb3f7ebca4f556732f7f7acc736 # Parent a05ada86fbe02c8bfa021bcf5d9cd56b2c626719 Added reporting functionality. diff -r a05ada86fbe0 -r 26c2cc5077aa README --- a/README Thu Aug 13 19:21:15 2009 +0300 +++ b/README Fri Aug 14 01:19:58 2009 +0300 @@ -1,4 +1,4 @@ -Malicious Attack Livid Termination Filter daemon (maltfilter) v0.7 +Malicious Attack Livid Termination Filter daemon (maltfilter) v0.8 ================================================================== Programmed by Matti 'ccr' Hämäläinen (C) Copyright 2009 Tecnic Software productions (TNSP) @@ -66,3 +66,4 @@ If you want to use the init script, you need to edit your init runlevel settings to enable it, for example in Debian/Ubuntu you can use rcconf(8) or chkconfig(8). + diff -r a05ada86fbe0 -r 26c2cc5077aa example.conf --- a/example.conf Thu Aug 13 19:21:15 2009 +0300 +++ b/example.conf Fri Aug 14 01:19:58 2009 +0300 @@ -1,6 +1,10 @@ -## Maltfilter configuration file. -## PLEASE READ THROUGH THIS FILE VERY CAREFULLY! +############################################################################# +### Maltfilter configuration file. +### PLEASE READ THROUGH THIS FILE VERY CAREFULLY! +############################################################################# +### General settings +############################################################################# # Verbosity level (0 = quiet, bigger values add noise. valid range 0 - 4) VERBOSITY = 4 @@ -10,14 +14,16 @@ # NOTICE! IF YOU DON'T CHANGE THIS TO 0, MALTFILTER WILL NOT DAEMONIZE! DRY_RUN = 1 -# Define system log files to scan. Only auth.log and Apache errorlog / -# common log format files are supported for now. You can have as many -# of SCANFILE settings as you wish. -SCANFILE = "/var/log/auth.log" -SCANFILE = "/var/log/httpd/error.log" -SCANFILE = "/var/log/httpd/access.log" +# Maltfilter logfile path and name (set empty "" if you don't want logging) +LOGFILE = "/var/log/maltfilter" + +# Full path to iptables binary +IPTABLES = "/sbin/iptables" +############################################################################# +### Actions, etc. settings +############################################################################# # Weeding treshold in hours. Entries older than this will be "weeded" # off from current netfilter settings. WEEDPERIOD = 72 @@ -30,6 +36,30 @@ # can use whatever rule chain name you want to here. ACTION = "DROP" +# IP addresses that should NOT be blocked under any circumstances. You should +# set this if you wish to have a surefire open channel from some host, even in +# the case someone tries to spoof IPs for denial of service. +# +# NOTICE! This setting supports only IPv4 addresses, no IPv6 or DNS names. +# You can have any number of NOBLOCK_IPS settings. +NOBLOCK_IPS = "192.121.86.15" +NOBLOCK_IPS = "74.125.45.100" + + +############################################################################# +### Logfiles +############################################################################# +# Define system log files to scan. Only auth.log and Apache errorlog / +# common log format files are supported for now. You can have as many +# of SCANFILE settings as you wish. +SCANFILE = "/var/log/auth.log" +SCANFILE = "/var/log/httpd/error.log" +SCANFILE = "/var/log/httpd/access.log" + + +############################################################################# +### Checks / tests +############################################################################# # Enabled checks (1 = enabled, 0 = disabled). Please read the test # descriptions from "check_log_line" function in the maltfilter script. CHK_SSHD = 1 @@ -43,17 +73,17 @@ # or that alternatively you have defined "safe" hosts in NOBLOCK_HOSTS below. CHK_ROOT_SSH_PWD = 0 -# Maltfilter logfile path and name (set empty "" if you don't want logging) -LOGFILE = "/var/log/maltfilter" -# Full path to iptables binary -IPTABLES = "/sbin/iptables" - -# IP addresses that should NOT be blocked under any circumstances. You should -# set this if you wish to have a surefire open channel from some host, even in -# the case someone tries to spoof IPs for denial of service. -# -# NOTICE! This setting supports only IPv4 addresses, no IPv6 or DNS names. -# You can have any number of NOBLOCK_IPS settings. -NOBLOCK_IPS = "192.121.86.15" -NOBLOCK_IPS = "74.125.45.100" +############################################################################# +### Reports +############################################################################# +# Define files for periodically updated status reports (refreshed once +# every few minutes.) Leave empty ("") if you do not want status reports. + +# Plain ASCII text file rerpot +STATUS_FILE_PLAIN = "" + +# HTML file and optional CSS stylesheet URL for the HTML +# (if left empty, no CSS is used) +STATUS_FILE_HTML = "" +STATUS_FILE_CSS = "" diff -r a05ada86fbe0 -r 26c2cc5077aa maltfilter --- a/maltfilter Thu Aug 13 19:21:15 2009 +0300 +++ b/maltfilter Fri Aug 14 01:19:58 2009 +0300 @@ -10,8 +10,9 @@ use Date::Parse; use Net::IP; +my $progversion = "0.8"; my $progbanner = -"Malicious Attack Livid Termination Filter daemon (maltfilter) v0.7\n". +"Malicious Attack Livid Termination Filter daemon (maltfilter) v$progversion\n". "Programmed by Matti 'ccr' Hamalainen \n". "(C) Copyright 2009 Tecnic Software productions (TNSP)\n"; @@ -19,14 +20,18 @@ ### Settings / configuration ############################################################################# my %settings = ( - "VERBOSITY" => 4, + "VERBOSITY" => 3, "DRY_RUN" => 1, - "WEEDPERIOD" => 72, + "WEEDPERIOD" => 150, "TRESHOLD" => 3, "ACTION" => "DROP", - "LOGFILE" => "/var/log/maltfilter", + "LOGFILE" => "", "IPTABLES" => "/sbin/iptables", + "STATUS_FILE_PLAIN" => "", + "STATUS_FILE_HTML" => "", + "STATUS_FILE_CSS" => "", + "CHK_SSHD" => 1, "CHK_KNOWN_CGI" => 1, "CHK_PHP_XSS" => 1, @@ -54,6 +59,10 @@ my %filehandles = (); my %hitcount = (); my %iplist = (); +my %reason = (); +my %reason_n = (); +my %ignored = (); +my %ignored_d = (); my $pid_file = ""; my $LOGFILE; @@ -67,13 +76,13 @@ # (1.1) Generic login scan attempts if ($merr =~ /^Failed password for invalid user \S+ from (\d+\.\d+\.\d+\.\d+)/) { - check_add_entry($1, $mdate, "SSHD", $settings{"CHK_SSHD"}); + check_add_entry($1, $mdate, "SSH login scan", "", $settings{"CHK_SSHD"}); } # (1.2) Root SSH login password bruteforcing attempts # NOTICE! Do not enable this setting, if you allow SSH root logins via # password authentication! Mistyping password may get you blocked then. :) elsif (/^Failed password for root from (\d+\.\d+\.\d+\.\d+)/) { - check_add_entry($1, $mdate, "Root SSH password bruteforce", $settings{"CHK_ROOT_SSH_PWD"}); + check_add_entry($1, $mdate, "Root SSH password bruteforce", "", $settings{"CHK_ROOT_SSH_PWD"}); } } # (2) Common/known exploitable CGI/PHP software scans (like phpMyAdmin) @@ -87,7 +96,7 @@ if ($merr =~ /^File does not exist: (.+)$/) { my $tmp = $1; if ($tmp =~ /\/mss2|\/pma|admin|sql|\/roundcube|\/webmail|\/bin|\/mail|xampp|zen|mailto:|appserv|cube|round|_vti_bin|wiki/i) { - check_add_entry($mip, $mdate, "CGI: $tmp", $settings{"CHK_KNOWN_CGI"}); + check_add_entry($mip, $mdate, "CGI vuln scan", $tmp, $settings{"CHK_KNOWN_CGI"}); } } } @@ -103,13 +112,13 @@ # used in the URIs. if ($merr =~ /\.php\?\S*?=http:\/\/([^\/]+)/) { if (!check_hosts($settings{"CHK_GOOD_HOSTS"}, $1)) { - check_add_entry($mip, $mdate, "PHP XSS: $merr", $settings{"CHK_PHP_XSS"}); + check_add_entry($mip, $mdate, "PHP XSS", $merr, $settings{"CHK_PHP_XSS"}); } } # (3.2) Try to match proxy scanning attempts elsif ($merr =~ /^http:\/\/([^\/]+)/) { if (!check_hosts($settings{"CHK_GOOD_HOSTS"}, $1)) { - check_add_entry($mip, $mdate, "Proxy scan: $merr", $settings{"CHK_PROXY_SCAN"}); + check_add_entry($mip, $mdate, "Proxy scan", $merr, $settings{"CHK_PROXY_SCAN"}); } } } @@ -125,12 +134,12 @@ my $msg = shift; if (defined($LOGFILE)) { print $LOGFILE "[".scalar localtime()."] ".$msg if ($settings{"VERBOSITY"} > $level); - } else { + } elsif ($settings{"DRY_RUN"}) { print $msg if ($settings{"VERBOSITY"} > $level); } } - +### Host and IP matching functions sub check_hosts_array($$) { my $chk_host = $_[1]; @@ -196,6 +205,10 @@ if (defined($iplist{$_[0]})) { mlog(2, "* Weeding $_[0] ($iplist{$_[0]})\n"); exec_iptables("-D", "INPUT", "-s", $_[0], "-d", "0.0.0.0/0", "-j", $settings{"ACTION"}); + undef($reason{$_[0]}); + undef($reason_n{$_[0]}); + undef($ignored{$_[0]}); + undef($ignored_d{$_[0]}); undef($iplist{$_[0]}); } } @@ -213,51 +226,239 @@ } } +### Output status file +sub cmp_ips($$) +{ + my @ipa = split(/\./, $_[0]); + my @ipb = split(/\./, $_[1]); + for (my $i = 0; $i < 4; $i++) { + return -1 if ($ipa[$i] > $ipb[$i]); + return 1 if ($ipa[$i] < $ipb[$i]); + } + return 0; +} + +sub cmp_ip_hits($$$$) +{ + return -1 if ($_[2] > $_[3]); + return 1 if ($_[2] < $_[3]); + return cmp_ips($_[0], $_[1]); +} + +sub printH($$$$) +{ + my $fh = $_[1]; + if ($_[0]) { + print $fh "".$_[3]."\n"; + } else { + my $c = ($_[2] <= 1) ? "=" : "-"; + print $fh $_[3]."\n". $c x length($_[3]) ."\n"; + } +} + +sub printTD($$$) +{ + my $fh = $_[1]; + if ($_[0]) { + print $fh "".$_[2].""; + } else { + print $fh $_[2]; + } +} + +sub printP($$$) +{ + my $fh = $_[1]; + if ($_[0]) { + print $fh "

\n".$_[2]."

\n"; + } else { + print $fh $_[2]."\n"; + } +} + +sub printElem +{ + my $fh = $_[1]; + if ($_[0]) { + print $fh $_[2]; + } elsif (defined($_[3])) { + print $fh $_[3]; + } +} + +sub bb($) +{ + return $_[0] ? "" : ""; +} + + +sub eb($) +{ + return $_[0] ? "" : ""; +} + +sub generate_status($$) +{ + my $filename = shift; + my $m = shift; + + return unless ($filename ne ""); + + mlog(-1, "dumping status '$filename'\n"); + open(STATUS, ">", $filename) or die("Could not open '".$filename."'!\n"); + my $f = \*STATUS; + my $mtime = scalar localtime(); + + printElem($m, $f, " + + + Maltfilter status report +"); + + printElem($m, $f, "") + if ($settings{"STATUS_FILE_CSS"}); + + printElem($m, $f, " + + +"); + + + printH($m, $f, 1, "Maltfilter v$progversion status report"); + printP($m, $f, "Generated: ".$mtime."\n"); + + printH($m, $f, 2, "Blocked"); + + printElem($m, $f, "\n". ""."\n"); + my $nexcluded = 0; + my $ntotal = 0; + foreach my $mip (sort { $hitcount{$b} <=> $hitcount{$a} } keys %iplist) { + $nexcluded++ if check_hosts_array(\@noblock_ips, $mip); + + printElem($m, $f, " "); + printTD($m, $f, sprintf("%-10d", $hitcount{$mip})); + printTD($m, $f, sprintf("%-15s", $mip)); + printElem(!$m, $f, " : "); + printTD($m, $f, scalar localtime($iplist{$mip})); + my @s = (); + foreach my $cond (sort keys %{$reason{$mip}}) { + push(@s, bb($m).$cond.eb($m)." [".$reason_n{$mip}{$cond}." hits] (".$reason{$mip}{$cond}.")"); + } + printTD($m, $f, join(", ".($m ? "
" : ""), @s)); + printElem($m, $f, "\n", "\n"); + $ntotal++; + } + printElem($m, $f, "
HitsIP-addressDate of last hitReason(s)
\n"); + printP($m, $f, bb($m).$ntotal.eb($m)." entries listed, ". + bb($m).($ntotal - $nexcluded).eb($m)." blocked, ".bb($m).$nexcluded.eb($m). + " excluded (defined in NOBLOCK_IPS).\n"); + + + printH($m, $f, 2, "All recorded hits in general"); + printP($m, $f, "List of 'hits' of suspicious activity noticed by Maltfilter, but not necessarily acted upon.\n"); + + printElem($m, $f, "\n". "" x 2 ."\n"); + my $hits = 0; + my @keys = sort { cmp_ip_hits($a, $b, $hitcount{$a}, $hitcount{$b}) } keys %hitcount; + my $nkeys = scalar @keys; + + my $printEntry = sub { + printTD($m, $f, sprintf("%-15s", $_[0])); + printElem(!$m, $f, " : "); + printTD($m, $f, sprintf("%-8d ", $_[1])); + }; + + my $kmax = $nkeys / 2; + for (my $i = 0; $i <= $kmax; $i++) { + printElem($m, $f, " "); + if ($i < $kmax) { + my $mip = $keys[$i]; + &$printEntry($mip, $hitcount{$mip}); + $hits += $hitcount{$mip}; + } + printElem(!$m, $f, " | "); + if ($i + $kmax + 1 < $nkeys) { + my $mip = $keys[$i + $kmax + 1]; + &$printEntry($mip, $hitcount{$mip}); + $hits += $hitcount{$mip}; + } + printElem($m, $f, "\n", "\n"); + } + + printElem($m, $f, "
IP-address# of hits
\n"); + + printP($m, $f, bb($m).(scalar keys %hitcount).eb($m)." IPs total, ".bb($m).$hits.eb($m)." hits total.\n"); + + + printH($m, $f, 2, "Ignored hit types"); + printP($m, $f, "List of hits that were ignored (not acted upon), because the test was disabled.\n"); + + printElem($m, $f, "\n\n"); + foreach my $mip (sort { cmp_ips($a, $b) } keys %ignored) { + printElem($m, $f, ""); + printTD($m, $f, sprintf("%-15s", $mip)); + printElem($m, $f, "", "\n"); + } + printElem($m, $f, "
IP-addressType (hits, last time of note)
", " : "); + foreach my $mcond (sort keys %{$ignored{$mip}}) { + my $s = $mcond." (".$hitcount{$mip}." hits, last ".scalar localtime($ignored_d{$mip}{$mcond}).")"; + unless ($ignored{$mip}{$mcond} eq "") { $s .= " for '".$ignored{$mip}{$mcond}."'"; } + print $f $s; + } + printElem($m, $f, "
\n"); + + printElem($m, $f, "\n\n"); + close(STATUS); +} + ### Check if given "try count" exceeds treshold and if entry ### is NOT in Netfilter already, then add it if so. -sub check_add_entry($$$$) +sub check_add_entry($$$$$) { my $mip = $_[0]; my $mdate = str2time($_[1]); - my $mreason = $_[2]; - my $mcond = $_[3]; + my $mclass = $_[2]; + my $mreason = $_[3]; + my $mcond = $_[4]; my $cnt = $hitcount{$mip}++; if ($cnt >= $settings{"TRESHOLD"} && check_time($mdate)) { my $pat; if (!$mcond) { - mlog(2, "* Ignoring $mip: $mreason\n"); + $ignored{$mip}{$mclass} = $mreason; + $ignored_d{$mip}{$mclass} = $mdate; return; } if (!defined($iplist{$mip})) { - if (!check_hosts_array(@noblock_ips, $mip)) { + if (!check_hosts_array(\@noblock_ips, $mip)) { # Add entry that has >= treshold hits and is not added yet mlog(1, "* Adding $mip ($mdate): $mreason\n"); exec_iptables("-I", "INPUT", "1", "-s", $mip, "-j", $settings{"ACTION"}); } $iplist{$mip} = $mdate; + $reason{$mip}{$mclass} = $mreason; + $reason_n{$mip}{$mclass}++; } else { # Over treshold, but is added, check if we can update the timedate - if ($iplist{$mip} >= 0) { - if ($mdate > $iplist{$mip}) { - $iplist{$mip} = $mdate; - } - } else { - # Empty date, set it now. - $iplist{$mip} = $mdate; + if ($iplist{$mip} < 0) { + $reason{$mip}{$mclass} = $mreason; + $reason_n{$mip}{$mclass}++; } + $iplist{$mip} = $mdate if ($mdate > $iplist{$mip}); } } } ### -### Utility functions +### Main helper functions ### +sub malt_init { + mlog(0, "Updating initial blocklist from netfilter.\n"); + update_iplist(-1); -sub malt_init { foreach my $filename (@scanfiles) { local *INFILE; - mlog(0, "- Parsing ".$filename." ...\n"); + mlog(0, "Parsing ".$filename." ...\n"); open(INFILE, "<", $filename) or die("Could not open '".$filename."'!\n"); $filehandles{$filename} = *INFILE; while () { @@ -265,13 +466,13 @@ check_log_line($_); } } - - mlog(0, "- Weeding out old entries.\n"); + + mlog(0, "Weeding old entries.\n"); weed_entries(); } sub malt_cleanup { - mlog(0, "- Closing open filehandles.\n"); + # Close open filehandles foreach my $filename (keys %filehandles) { close($filehandles{$filename}); } @@ -279,8 +480,8 @@ sub malt_scan { ### Keep on reading - mlog(1, "- Entering main scanning loop.\n"); - my $counter = 0; + mlog(1, "Entering main scanning loop.\n"); + my $counter = -1; while (1) { my %filepos = (); foreach my $filename (keys %filehandles) { @@ -289,15 +490,17 @@ check_log_line($_); } } - sleep(5); - if ($counter++ >= 5) { + if ($counter < 0 || $counter++ >= 30) { # Every once in a while, update known IP list from iptables # (in case entries have appeared there from "outside") # and perform weeding of old entries. $counter = 0; update_iplist(time()); weed_entries(); + generate_status($settings{"STATUS_FILE_PLAIN"}, 0); + generate_status($settings{"STATUS_FILE_HTML"}, 1); } + sleep(5); foreach my $filename (keys %filehandles) { seek($filehandles{$filename}, $filepos{$filename}, 0); } @@ -322,7 +525,7 @@ } sub malt_term { - mlog(-1, "Receinved TERM, quitting.\n"); + mlog(-1, "Received TERM, quitting.\n"); malt_cleanup(); malt_finish(); exit(1); @@ -348,13 +551,23 @@ my $argc = $#ARGV + 1; if ($argc < 1) { print $progbanner. - "\nUsage: maltfilter [config filename]\n"; + "\n". + "Usage: maltfilter [config filename]\n". + " maltfilter -f [config filename]\n". + "-f turns on the full report mode.\n"; exit; } # Test pid file existence +my $report; $pid_file = shift; -die("'$pid_file' already exists, not starting.\nIf the daemon is NOT running, remove the pid-file and re-start.\n") if (-e $pid_file); +if ($pid_file eq "-f") { + $report = 1; +} else { + die("'$pid_file' already exists, not starting.\n". + "If the daemon is NOT running, remove the pid-file and re-start.\n") + if (-e $pid_file); +} # Read configuration file if (defined(my $config_file = shift)) { @@ -399,6 +612,11 @@ die("Errors in configuration file '$config_file', bailing out.\n") unless ($errors == 0); } +# Force dry run mode if we are reporting only +if ($report) { + $settings{"DRY_RUN"} = 1; +} + # Clean up certain arrays duplicate entries my %saw = (); @scanfiles = grep(!$saw{$_}++, @scanfiles_def); @@ -428,13 +646,19 @@ mlog(-1, "Not blocking following IPs: ".join(", ", @noblock_ips)."\n"); # Initialize -update_iplist(-1); malt_init(); # Fork to background, unless dry-running if ($settings{"DRY_RUN"}) { - malt_scan(); - malt_cleanup(); + if ($report) { + mlog(-1, "Outputting report files.\n"); + generate_status($settings{"STATUS_FILE_PLAIN"}, 0); + generate_status($settings{"STATUS_FILE_HTML"}, 1); + malt_cleanup(); + } else { + malt_scan(); + malt_cleanup(); + } } else { if (my $pid = fork) { open(PIDFILE, ">", $pid_file) or die("Could not open pid file '".$pid_file."' for writing!\n");