Migrate FreeBSD to Xen

There seems to be a lot of tutorials with respect to how you can dump/restore FreeBSD implementations. However, none of them appear to be all encompassing what is actually required from start to finish during the entire process.

The one thing that I think is lacking in proper documentation is utilizing FreeBSD in a LiveCD scenario (LiveFS) within a network capacity (necessary for migration).

We decided to write this tutorial so that people could have one place to establish all the necessary things required for this type of migration from start to finish.

In this scenario we actually migrated a FreeBSD implementation on VMWARE to XEN HVM. In the end, there were no technical problems with FreeBSD actually running after it was migrated — it ran beautifully actually.

I should note that this was tested with FreeBSD 7.2-RELEASE disc images.

Please find the guide below :

Prepare OLD Instance

1. Boot into old operating system

2. Take note of partition slices / slice names / sizes / etc

3. Reboot with FreeBSD LiveFS disc

Prepare NEW Xen

1. Boot Xen instance with FreeBSD Disc 1 ISO

2. Partition / install boot loader exactly the same slices as the old instance. To be extra careful, give your slices a bit more disc space than the old implementation.

3. Write changes & reboot with FreeBSD LiveFS disc

Establish FreeBSD LiveFS environment

You need to establish a few things to get SSH / DUMP / RESTORE to work properly on both the ”’old”’ and ”’new”’ instances

1. Boot into FreeBSD LiveFS (Fixit > livefs)

2. Create the following folders :

/etc/ssh
/usr/sbin
/usr/bin
/root
/root/.ssh

3. Copy the following files :

cp /mnt2/bin/ps /bin
cp /mnt2/sbin/sysctl /sbin
cp /mnt2/etc/ssh/* /etc/ssh
cp /mnt2/bin/csh /bin
cp /mnt2/bin/cat > /bin
cp /mnt2/sbin/restore > /sbin

4. Set an IP address on both old and new instances:

new :

ifconfig eth0 10.0.0.50 netmask 255.255.255.0

old :

ifconfig eth0 10.0.0.60 netmask 255.255.255.0

5. Start sshd :

/mnt2/etc/rc.d/sshd forcestart

Start transferring slices

1. To allow for transferring of partitions properly, the /tmp partition should be mounted on the new Xen instance :

mount -t ufs /dev/ad0s1e /tmp

2. For the first partition you wish to transfer, mount the empty slice on the new xen instance :

mount -t ufs /dev/ad0s1a /mnt/ufs.1

Sometimes you have to fsck mark the filesystem clean to mount it :

fsck /dev/ad0s1a

3. On the old instance :

dump -0aLf - /dev/ad0s1a | ssh 10.0.0.50 "cd /mnt/ufs.1 && cat | restore -rf -"

That should dump/restore the slice from old > new.

Final things on the new Xen instance

Dont forget to boot the new instance in single user mode and modify ”’fstab”’ to reflect the new slice names (if applicable), as well as ”’rc.conf”’ for any hard coded interface names, etc. FreeBSD won’t boot if the right slice names / interface names aren’t present. Or at least cause problems.

You can mount the /etc slice while still in the LiveFS for the new FreeBSD instance.

Hopefully this was helpful! Obviously this has nothing to do with Xen, other than the fact that we were migrating the FreeBSD vmware instance to Xen.

You can do this on “real” machines, or from xen to vmware or anywhere. As long as the hardware is compatible.

1 Comment

Amazon S3 Backup script with encryption

With the advent of cloud computing, there have been several advances as far as commercial cloud offerings, most notably Amazon’s EC2 computing platform as well as their S3 Storage platform.

Backing up to Amazon S3 has become a popular alternative to achieving true offsite backup capabilities for many organizations.

The fast data transfer speeds as well as the low cost of storage per gigabyte make it an attractive offer.

There are several free software solutions that offer the ability to connect to S3 and transfer files. The one that shows the most promise is s3sync.

There are already a few guides that show you how to implement s3sync on your system.

The good thing is that this can be implemented in Windows, Linux, FreeBSD among other operating systems.

We have written a simple script that utilizes the s3sync program in a scheduled offsite backup scenario. Find our script below, and modify it as you wish. Hopefully it will help you get your data safely offsite ;)

#!/bin/sh
# OffSite Backup script

currentmonth=`date "+%Y-%m-%d %H:%M:%S"`

export AWS_ACCESS_KEY_ID="YOUR-ACCESS-KEY"
export AWS_SECRET_ACCESS_KEY="YOUR-SECRET-ACCESS-KEY"

echo "Offsite Backup Log: " $currentmonth > /var/log/offsite-backup.log
echo -e "----------------------------------------" >> /var/log/offsite-backup.log
echo -e "" >> /var/log/offsite-backup.log

# Archive Files and remove files older than 3 days
/usr/bin/find /home/offsite-backup-files -type f -mtime +3 -delete

# Compress and archive a few select key folders for archival and transfer to S3
tar -czvf /home/offsite-backup-files/offsite-backup-`date "+%Y-%m-%d"`.tar.gz /folder1 /folder2 /folder3 >> /var/log/offsite-backup.log 2>&1

# Transfer the files to Amazon S3 Storage via HTTPS
/usr/local/bin/ruby /usr/local/bin/s3sync/s3sync.rb --ssl -v --delete -r /home/offsite-backup-files your-node:your-sub-node/your-sub-sub-node >> /var/log/offsite-b
ackup.log 2>&1

# Some simple error checking and email alert logging
if [ "$?" -eq 1 ]
then
        echo -e "***OFFSITE BACKUP JOB, THERE WERE ERRORS***" >> /var/log/offsite-backup.log 2>&1
        cat /var/log/offsite-backup.log | mail -s "Offsite Backup Job failed" you@yourdomain.com
        exit 1
else
        echo -e "Script Completed Successfully!" >> /var/log/offsite-backup.log 2>&1
        cat /var/log/offsite-backup.log | mail -s "Offsite Backup Job Completed" your@yourdomain.com
        exit 0
fi

Now if your data happens to be sensitive (most usually is), usually encrypting the data during transit (with the –ssl flag) is not enough.

You can encrypt the actual file before it is sent to S3, as an alternative. This would be incorporated into the tar command with the above script. That line would look something like this :

/usr/bin/tar -czvf - /folder1 /folder2 /folder3 | /usr/local/bin/gpg --encrypt -r you@yourdomain.com > /home/offsite-backup-files/offsite-backups-`date "+%Y-%m-%d"`.tpg

Alternative to gpg, you could utilize openssl to encrypt the data.

Hopefully this has been helpful!

2 Comments

Compress files and folders over the network without using rsync

The following command ssh’s to your remote server, tar + gzips a directory, and then outputs the compressed stream to your local machine.

This is a good alternative to rsync. Even though rsync can compress the transfer mid stream, the receiving end is still the un-extracted copy.


ssh -l username 0.0.0.0 '(cd /home/mysql-backups/ && tar -czf - . -C /home/mysql-backups)' >> test.tar.gz 2>&1

To do the above command, and extract it on your end (after transferring the compressed file over the network), simply do the following :


ssh -l username 0.0.0.0 '(cd /home/mysql-backups/ && tar -czf - . -C /home/mysql-backups)' | tar -xzf -

These commands could theoretically incorporate pgp encryption to encrypt and compress the archive before it travels across the network, for increased security. That is why this alternative to rsync may be preferential to some.

Obviously you could locally encrpyt + compress , then rsync, but its always a good idea to not utilize local storage for this process and keep all the storage capacity on the centralized storage system that you have already allocated.

No Comments

Script to distribute SSH Keys across many servers

Hello once again!

You may remember an earlier post that detailed how to implement SSH Key based authentication.

We believe it is important, when administering many (sometimes hundreds or thousands) of servers, to implement a strategy that can allow systems administrators to seamlessly run scripts, system checks or critical maintenance across all the servers.

SSH Key authentication allows for this potential. It is a very powerful strategy and should be maintained and implemented with security and efficiency as a top priority.

Distributing keys for all authorized systems administrators is something that would allow for the maintenance of this authentication system much easier — when an admin leaves or is dismissed, you need to be able to remove his or her’s keys from the “pool” quickly.

The idea behind this script is to have a centralized, highly secure and restricted key repository server. Each server in your environment would run this script to “pull” the updated key list from the central server. The script would run as a cron job and can run as often as you like. Ideally every 5-10 minutes would allow for quick key updates / distribution.

Here is the perl script :

#!/usr/bin/perl
#
# A script to sync ssh keys on UNIX servers automatically.  This
# will not overwrite user installed ssh keys
# Star Dot Hosting : www.stardothosting.com
#

use strict;
use IPC::Open3;
use File::Copy;

use POSIX ":sys_wait_h";

# This is overkill but FreeBSD may install wget in
# /usr/local/bin in some cases.
$ENV{PATH} = "/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin";

####################################################

use constant URL => 'http://keys.keyhostserver.com/ssh.keys.txt';
use constant WGET => 'wget -q -O - ';
use constant KEYS_FILE => '/root/.ssh/authorized_keys';
use constant RESTRICTED => 'http://keys.keyhostserver.com/restricted.txt';

####################################################

my ($url, $wget, $keys_file, $restricted, %restrict);

for (my $i=0;$i) {
                chomp;
                $restrict{$_}++;
        }
}

$pid = open3(\*WTR, \*RTR, \*ERR, "$wget");

while () {
        next unless $_ =~ / (\S+)\@company$/;
        next if $restrict{$1};
        $company_keys .= $_;
}

$user_keys = read_key_file();

# Sanity check
my @rows = split('\n', $company_keys);

if (scalar @rows < 5) {
        print "Less than 5 company keys found, not installing keys..\n";
        exit(1);
} elsif ($company_keys !~ /backup\@company\n/) {
        print "No backup key found, exiting..\n";
        exit(1);
}

open(TMP, ">$keys_file.$$.tmp") or die "Could not open tmp keys file: $!\n";
print TMP $company_keys;
print TMP $user_keys;
close(TMP);

# Sanity check

my (undef,undef,undef,undef,undef,undef,undef,$size,undef,undef,undef,undef,undef) = stat("$keys_file.$$.tmp");

if ($size < 5000) {
        print "Keys file less than 5k, not writing";
        exit(1);
}

move("$keys_file.$$.tmp", $keys_file);

sub read_key_file {
        my $user_buf;

        open(KEY_FILE, "< $keys_file") or die "Could not open ssh key file; $!\n";

        while () {
                next if $_ =~ /company$/;
                $user_buf .= $_;
        }

        close(KEY_FILE);
        return($user_buf);
}

sub sig_chld {
        my $pid = waitpid(-1, WNOHANG);
}

sub usage {
        print STDERR <<"EOS";

        Usage: $0 -[kuh]

                -k        Keys file to write to (default: @{[KEYS_FILE]})
                -u         URL to download keys from (default: @{[URL]})
                -h              This screen

EOS
        exit(1);
}

1;

__END__

Note that it downloads the public keys via http with wget. This can be easily modified to utilize https, if necessary, or perhaps even another protocol to make the transfer. HTTP Was chosen because the public keys are harmless and http is the easiest method. HTTPS would be desirable, however.

We hope this script helps you along the way towards making your life easier! ;)

No Comments

Setup Up Exim with ClamAV and Spamassassin

I decided to post this article on implementing a simple single mail server with anti-spam and anti-virus capabilities.

This guide hopefully will help you on your way to configuring a basic mail system on Linux (specifically Debian).

Installing and configuring Exim 4 on Debian

1. First, install all the necessary Debian packages are on the system as the root user. (The exim4 package will REPLACE the exim package.)

NOTE: If you are using the stable branch, it is suggested to use the debian volatile packages (along with the security packages) so that your system is using the most up-to-date critical packages (like ClamAV) for security purposes. For production servers, you may not want to run a mixed stable/testing/unstable system (though I know some of you do!). To use these packages, see http://volatile.debian.net/ for more information. For those of you who are impatient and don’t want to find the correct mirror, here’s is what I added to my /etc/apt/sources.list file:

deb http://volatile.debian.net/debian-volatile sarge/volatile main contrib

I used aptitude to install these packages, but you could also use the old apt-get method:

apt-get install clamav-daemon \
clamav-freshclam exim4-daemon-heavy exim4 \
courier-base courier-authdaemon courier-imap \
courier-pop spamassassin wget spamc sa-exim

When going through the exim4 config, be sure to select the multiple file configuration layout. If you didn’t (or weren’t prompted for it), simply set dc_use_split_config to true in the /etc/exim4/update-exim.conf.conf file. (Thanks Mike!)

2. Create your Maildir directory

maildirmake ~/Maildir/

3. Now we want to make exim4 use Maildir format mailboxes. Modify the file /etc/exim4/update-exim4.conf.conf so that it contains:

dc_localdelivery='maildir_home'

4. We need to Edit /etc/default/spamassassin to enable spamd.

5. Each user can set up their own filters by creating a .forward file in their home directory. If the first line of this file reads

# Exim filter then Exim4 will treat it as a filter.

Here is an example of an Exim filter that checks the headers that SpamAssassin adds and puts the mail in the appropriate Maildir folder:

      # Exim filter
      if $h_X-Spam-Status: CONTAINS "Yes"
           or
        $h_X-Spam-Flag: CONTAINS "Yes"
      then
        save $home/Maildir/.Spam/
        finish
      endif

Exim’s Interface To Mail Filtering (PDF format) – Local copy

6. Many system administrators like to set up the Maildir directories and .forward filter file in the /etc/skel directory so that when they make a new user on the system, everything is automatically copied over. I suggest that you do this as well as it makes things easier.

7. Before going live with the mail server, we will want to test it!

Testing the implementation

1. Generate the new configuration:

update-exim4.conf

If you made it through this, then your config files don’t have any syntax errors.

exim4 -bV

If that works, then there are no config issues

2. Next, start exim by issuing:

/etc/init.d/exim4 start

Above assumes that you are running exim4 as a daemon, and not through inetd

3. Now, check a local address:

            exim4 -bt local_user@example.com

4. Check sending an email:

            exim4 -v mailbox_you_can_check@dom.ain
               From: user@your.domain
               To: mailbox_you_can_check@dom.ain
               Subject: Testing exim

               Testing exim
               .

You should now see some messages to let you know that the email was sent or information about what went wrong.

5. To test with full debug output using a specific config file, use something like:

            exim4 -C /etc/exim/exim_example.conf -d -bt user@example.com

6. To test the config coming from a specified ip address, use:

            exim4 -bh 192.168.1.10

            HELO example.com
               MAIL FROM: 
               RCPT TO: 
               DATA
               Subject: something
               your message here
               .
               QUIT

8. Add the following to your /etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs file:

      # This tells what virus scanner to use
      av_scanner = clamd:/var/run/clamav/clamd.ctl

9. Edit /etc/exim4/conf.d/acl/40_exim4-config_check_data to inlude the following before the “# accept otherwise” line:

      # Reject messages that have serious MIME errors.
         # This calls the demime condition again, but it
         # will return cached results.
         deny message = Serious MIME defect detected ($demime_reason)
         demime = *
         condition = ${if >{$demime_errorlevel}{2}{1}{0}}

         # Reject file extensions used by worms.
         # Note that the extension list may be incomplete.
         deny message = This domain has a policy of not accepting certain types of attachments \
                        in mail as they may contain a virus.  This mail has a file with a .$found_extension \
                        attachment and is not accepted.  If you have a legitimate need to send \
                        this particular attachment, send it in a compressed archive, and it will \
                        then be forwarded to the recipient.
         demime = exe:com:vbs:bat:pif:scr

         # Reject messages containing malware.
         deny message = This message contains a virus ($malware_name) and has been rejected
         malware = *

10. Then, you need to enable ClamAV.

a) Firstly, you will want to be sure that it is running against messages. In /etc/exim4/sa-exim.conf, search for SAEximRunCond:

SAEximRunCond: ${if and {{def:sender_host_address} {!eq {$sender_host_address}{127.0.0.1}} {!eq {$h_X-SA-Do-Not-Run:}{Yes}} } {1}{0}}

That is simply skipping the scan on anything from the local machine or if the X-SA-Do-Not-Run header in the message is set to Yes. If you just want exim to run ClamAV on all messages, use this:

SAEximRunCond: 1

b) Before restarting ClamAV, we need to be sure that all of the access rights are in place so that the scans actually happen. The best way to handle this is to add the clamav user to the Debian-exim group. Either manually edit /etc/group, or simple run:

adduser clamav Debian-exim

c) Be sure that /etc/clamav/clamd.conf contains a line that reads:

AllowSupplementaryGroups

d) Set the file permissions for the /var/run/clamav directory to allow for the correct user to use it:

            chown Debian-exim.Debian-exim /var/run/clamav
            chmod g+w /var/run/clamav

e) A restart of ClamAV is necessary for the changes to take effect:

/etc/init.d/clamav-daemon restart

11. You should now be able to get your mail via IMAP with a mail client like Mozilla.

Check your headers (View Source) and see that SpamAssassin has added its headers. SMTP-end virus scanning should also be taking place. Check your /var/log/clamav/clamav.log to monitor this.

Multiple Domain Alias Files

The steps below are used to enable support for having multiple virtual domains each with its own alias file.

1. Exim will need to have the alias files for each domain.

a) Create the /etc/exim4/virtual directory.
b) For each virtual domain, create a file that contains the aliases to be used named as the domain.

For example, if I example.com was one of my domains, I’d do the following:

a) Create the /etc/exim4/virtual/example.com file.
b) If my system users were sys1, sys2, and sys3, and their email addresses were to be joe, john, jason, I’d put the following into the domain alias file:

                  joe:    sys1@localhost
                  john:   sys2@localhost
                  jason:  sys3@localhost

If john was also to get all mail addressed to info@example.com, you would add this entry:

info:   sys2@localhost

If you wanted all mail to user1@example.com to go to another email account outside of this domain, you would enter:

user1:  a.user@some.domain

If you wanted all mail directed at any address other than what is defined in the alias file to go to joe, you’d enter:

*:      sys1@localhost

In the above examples, the “@localhost” suffix to the user names forces the delivery to a system user. I found that if you do not include this in the alias files and your machine’s host name is within one of the domains handled by exim, every system user would need an entry in the machine’s domain in order to be delivered corectly.

For instance, if your host name was mail.example1.com and example1.com was handled by this server this would be needed. This would allow delivery to all the system user names at example1.com.

The reason is simple, and I will try to illustrate it for you here:

a) exim receives a message delivered to joe.blow@example3.com
b) The alias file for this domain has joe.blow: jblow in it.
c) This would translate to jblow@domain-of-the-system
d) The process would be repeated using jblow@domain-of-the-system
e) If there was no entry in the domain-of-the-system alias file for jblow, the message would be undeliverable (or non-routable)

You could even have special redirects like the following:

script: "| /path/to/some/script"
prev:   :fail: $local_part left!
kill:   :blackhole:

2. Edit /etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs by replacing the current local_domains line with:

domainlist local_domains = @:localhost:dsearch;/etc/exim4/virtual

3. Create /etc/exim4/conf.d/router/350_exim4-config_vdom_aliases with the following content:

            vdom_aliases:
            driver = redirect
            allow_defer
            allow_fail
            domains = dsearch;/etc/exim4/virtual
            data = ${expand:${lookup{$local_part}lsearch*@{/etc/exim4/virtual/$domain}}}
            retry_use_local_part
            pipe_transport   = address_pipe
            file_transport   = address_file

4. Now, regenerate your exim4 config:

update-exim4.conf

5. If there were no errors, restart exim4:

/etc/init.d/exim4 restart

Domain Dependent Maximum Message Size

The next step for my server is to give each domain a configurable message size limit. Then, when the server get’s a message that is larger than the target domain’s size limit, I want to send a message back to the original sender telling them why the message was not delivered. However, I also want to have that message customized for each domain. That way, the domain owners can provide detailed instructions on how to send large messages to their domain if it is necessary. Of course, there will also need to be some kind of default size limit and message for domains that do not need the customization.

1. Create /etc/exim4/domain-size-limits to contain the list of domains and their maximum message size limits. You can also add a wildcard at the end entry if you want to set a default limit. The file may look something like the following:

      example.com: 20M
      example1.com: 5M
      *: 15M

This provides you a quick way to edit the values. The values will also take effect as soon as the file is saved – no need to restart exim!

2. OK, now we know what domains we want to customize the size for. Now it’s time to create a message to send for those domains. Create /etc/exim4/domain-size-limit-messages with content similar to:

      exmaple.com: The largest acceptable message size for Example.com is\
                   ${expand:${lookup{$domain}lsearch*@{/etc/exim4/domain-size-limits}}}.\
                   Your message was $message_size. If you feel that $local_part@$domain\
                   should really get your message, then visit http://www.example.com/files/\
                   where you can upload any large files. If you select $local_part@$domain\
                   from the "notify" list, they will receive a message with a link directly\
                   to your file.
      *:           The largest acceptable message size for $domain is\
                   ${expand:${lookup{$domain}lsearch*@{/etc/exim4/domain-size-limits}}}.\
                   Your message size was $message_size. Please revise your message so it\
                   does not exceed this maximum file size and resend. If this is not\
                   possible, contact the recipient in another way.

As you see, we have one domain that has a custom message sent out, and have defined a default message for all other domains. These messages can be edited at any time and do not need an exim restart to take effect.

3. Now for the fun part! We need a way to catch the messages that are too large for the domain! First, create /etc/exim4/conf.d/router/325_exim4-config_large_messages with the following:

      large_messages:
          driver = accept
          domains = dsearch;/etc/exim4/virtual
          condition = ${if >{$message_size}{${expand:${lookup{$domain}lsearch*@{/etc/exim4/domain-size-limits}}}} {yes}{no}}
          transport = bounce_large_messages
          no_verify

This router dynamically checks which domains are available and what their limits are set to.

4. Now create /etc/exim4/conf.d/transport/40_exim4-config_bounce_large_messages with the following content:

      # This bounces a message to people who send files too large for that domain
      bounce_large_messages:
        driver = autoreply
        from = $local_part@$domain
        to = $sender_address
        subject = Re: ${escape:$h_subject:}
        text = ${expand:${lookup{$domain}lsearch*@{/etc/exim4/domain-size-limit-messages}}}

This transport then sends the original sender a message using the text looked up from the domain-size-limit-messages file for that domain. The From: field is filled in with the intended recipient of the message – appearing to be a reply.

This was actually very simple to put together once I realized what I needed to do. The above is based on what I found in the Exim FAQ

Configuration Tips

Maybe this is something I should have said in the beginning, but at the time or writing this document, I had never set up an exim4 server, and the only exim3 server I had was used with the default debconf install. Therefore, if you see something on this page that could be done in a more elegant, more efficient or just plain better way, please send me a note.

3 Comments

ProFTPD with MySQL Authentication

Since this setup uses one FTP account to create user home directories and upload files, a compromise to this FTP user would cause the attacker to gain access to all FTP user home directories. I guess it just depends on how much you trust the DefaultRoot directive in Proftpd. I run Proftpd in its own chroot environment in addition to using DefaultRoot, so I’m used to feeling pretty safe with my Proftpd install. Anyway, here’s how I did the install/configuration

1. install proftpd-mysql from the ports with WITH_QUOTA set:

cd /usr/ports/ftp/proftpd-mysql/
env WITH_QUOTA=yes make
env WITH_QUOTA=yes make install

2. Add the global proftpd user & Proftpd group to your system.

I used uid & gid 5500 simply because that’s what was used at one of the sites I was referencing (listed below).

pw groupadd -n Proftpd -g 5500
pw useradd proftpd -u 5500 -g Proftpd -s /sbin/nologin -d /dev/null -c "Proftpd User"

3. Create the mySQL database

create database proftpd;
grant all on proftpd.* to 'proftpd'@'localhost' identified by 'password'

( change ‘password’ to something secret! )

4. Create the mySQL tables for the users & quota

create table proftpdUsers (

sqlUID int unsigned auto_increment not null,
userName varchar(30) not null unique,
passwd varchar(80) not null,
uid int unsigned not null unique,
gid int unsigned not null,
homedir tinytext,
shell tinytext,
primary key(sqlUID)

) ;

create table proftpdGroups (

sqlGID int unsigned auto_increment not null,
groupName varchar(30) not null unique,
gid int unsigned not null unique,
members tinytext,
primary key(sqlGID)
);

CREATE TABLE proftpdQuotaLimits (
name VARCHAR(30),
quota_type ENUM("user", "group", "class", "all") NOT NULL,
per_session ENUM("false", "true") NOT NULL,
limit_type ENUM("soft", "hard") NOT NULL,
bytes_in_avail FLOAT NOT NULL,
bytes_out_avail FLOAT NOT NULL,
bytes_xfer_avail FLOAT NOT NULL,
files_in_avail INT UNSIGNED NOT NULL,
files_out_avail INT UNSIGNED NOT NULL,
files_xfer_avail INT UNSIGNED NOT NULL
);

CREATE TABLE proftpdQuotaTallies (
name VARCHAR(30) NOT NULL,
quota_type ENUM("user", "group", "class", "all") NOT NULL,
bytes_in_used FLOAT NOT NULL,
bytes_out_used FLOAT NOT NULL,
bytes_xfer_used FLOAT NOT NULL,
files_in_used INT UNSIGNED NOT NULL,
files_out_used INT UNSIGNED NOT NULL,
files_xfer_used INT UNSIGNED NOT NULL
);

5. Add a test user to the proftpd database

(assumes /home/ftp is where you keep your ftp users. Otherwise, change the homedir location). This is certainly not a necessary step, but you should probably check to see if your configuration is working. You can delete this user later.

insert into proftpdUsers values ( 0, 'test', 'test', 5500, 5500, '/home/ftp/test', '/sbin/nologin' );

6. Set your proftpd configuration to use the mySQL authentication and quotas:

(NOTE: this is not a complete configuration file, it’s basically just the default config file with mySQL auth & quotas added, but note that the User and Group directives are the user & group we added in step 2. )

MaxInstances 30

# Set the user and group under which the server will run.
User proftpd
Group Proftpd

# To cause every FTP user to be "jailed" (chrooted) into their home
# directory, uncomment this line.
DefaultRoot ~

# Normally, we want files to be overwriteable.
AllowOverwrite on

# Bar use of SITE CHMOD by default

DenyAll


# Log format and location
LogFormat               default "%t %h %a %s %m %f %b %T \"%r"\"
ExtendedLog             /var/log/proftpd.log ALL default
SystemLog               /var/log/proftpd.log ALL default
TransferLog             /var/log/proftpd.log ALL default

# Uncomment this if you have "invalid shell" errors in your proftpd.log
#RequireValidShell       off

# The passwords in MySQL are encrypted using CRYPT
SQLAuthTypes Plaintext Crypt
SQLAuthenticate users* groups*

# used to connect to the database
# databasename@host database_user user_password
SQLConnectInfo proftpd@localhost proftpd yourdatabasepassword

# Here we tell ProFTPd the names of the database columns in the "usertable"
# we want it to interact with. Match the names with those in the db
SQLUserInfo proftpdUsers userName passwd uid gid homedir shell

# Here we tell ProFTPd the names of the database columns in the "grouptable"
# we want it to interact with. Again the names match with those in the db
SQLGroupInfo proftpdGroups groupName gid members

# set min UID and GID - otherwise these are 999 each
SQLMinID 5000

#============
# User quotas
# ===========
QuotaEngine on
QuotaDirectoryTally on
QuotaDisplayUnits Mb
QuotaShowQuotas on

# create a user's home directory on demand if it doesn't exist
SQLHomedirOnDemand on

SQLNamedQuery get-quota-limit SELECT "name, quota_type, per_session, limit_type, bytes_in_avail, bytes_out_avail, bytes_xfer_avail, files_in_avail, files_out_avail, files_xfer_avail FROM proftpdQuotaLimits WHERE name = '%{0}' AND quota_type = '%{1}'"

SQLNamedQuery get-quota-tally SELECT "name, quota_type, bytes_in_used, bytes_out_used, bytes_xfer_used, files_in_used, files_out_used, files_xfer_used FROM proftpdQuotaTallies WHERE name = '%{0}' AND quota_type = '%{1}'"

SQLNamedQuery update-quota-tally UPDATE "bytes_in_used = bytes_in_used + %{0}, bytes_out_used = bytes_out_used + %{1}, bytes_xfer_used = bytes_xfer_used + %{2}, files_in_used = files_in_used + %{3}, files_out_used = files_out_used + %{4}, files_xfer_used = files_xfer_used + %{5} WHERE name = '%{6}' AND quota_type = '%{7}'" proftpdQuotaTallies

SQLNamedQuery insert-quota-tally INSERT "%{0}, %{1}, %{2}, %{3}, %{4}, %{5}, %{6}, %{7}" proftpdQuotaTallies

QuotaLimitTable sql:/get-quota-limit
QuotaTallyTable sql:/get-quota-tally/update-quota-tally/insert-quota-tally

1 Comment

Generate a self-signed SSL Certificate with OpenSSL

Occasionally it may be necessary to generate a self-signed SSL certificate. This could be for internal websites, or for other internal uses that may require secure encrypted network transmissions.

Generating a self-signed certificate may be an easy task for the intermediate or senior level admin, however we decided to post this guide for everyone to use, since using the guide as a reference may hopefully be useful to those of you out there ;)

1. Generate an SSL key without a passphrase, enter:

openssl genrsa -out /etc/httpd/ssl/mycorp.com.key 1024

2: Create a self-signed certificate, enter:

openssl req -new -key /etc/httpd/ssl/mycorp.com.key -x509 -out /etc/httpd/ssl/mycorpcom.crt -days 999

Sample output:

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:CA
Locality Name (eg, city) []:CA
Organization Name (eg, company) [Internet Widgits Pty Ltd]:mycorp, LLC
Organizational Unit Name (eg, section) []:Sales
Common Name (eg, YOUR name) []:
Email Address []:you@mycorp.com

My Sample Apache httpd.conf virtual host file:

DocumentRoot "/var/www/html/ssl_doc_root/"
ServerAdmin you@mycorp.com
ServerName www.mycorp.com
SSLEngine On
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:
+EXP:+eNULL
SSLCertificateFile /etc/httpd/ssl/mycorpcom.crt
SSLCertificateKeyFile /etc/httpd/ssl/mycorp.com.key
SetEnvIf User-Agent ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0

3. Restart httpd/Apche:

service httpd restart

Thats it! Test the SSL Cert to ensure it loads fine and reflects the proper values when you examine the properties.

2 Comments

Manage Nagios with Scripts

Working at many different organisations over the past 10 years, I have been involved in the implementation and maintenance of many different monitoring implementations. These include commercial and open source implementations, such as :

- Nagios
- IP Monitor
- Uptime
- OpenNMS
- Zabbix

Although Nagios may not be the most scalable or dynamic solution, for some organisations that perhaps have 1-100 servers, Nagios may be the best solution.

Additionally, the ability to write custom plugins, as well as the inherent SSL / TLS encryption of the NRPE checks, it may be the most viable. There are pro’s and con’s for each solution out there, and it is completely dependant on the skill level, nature of environment and available time for management / maintenance.

During the course of utilising Nagios, we noticed that one of the most time consuming tasks was maintaining the flat file configuration for adding, removing and modifying hosts within Nagios.

As a result, it was decided to write a quick Perl based script to manage the day-to-day tasks of adding and removing hosts within Nagios. When all is said and done, it really does save ALOT of time. This script can be integrated with existing control based management situations or other automation scripts / solutions where command line options and external scripting / plugins are possible. This way, you can encompass a more rounded, standardised and reliable way of managing your systems in Nagios.

In order for the script to work, you need to have 3 types of servers :

- Windows
- Unix/Linux
- VPS (Virtual Private Server)

Obviously you can modify the script to encompass an unlimited number of categories. Basically the script has defined three pre-existing hosts in the nagios hosts.cfg / hostgroups.cfg and services.cfg files to model them when adding the new server, based on your input.

Please take a look at the script, hopefully it will help make your life a little easier! ;)

#!/usr/bin/perl

# Don't break me, I'm used by automated scripts.

###############################################################################
# Star Dot Hosting : www.stardothosting.com
# Nagios Config Manager
# Description: This program will add/remove entries from nagios.
# The files will be backed up in a archive before any changes are made.
###############################################################################
# Perl Libraries
use File::Copy;
use Switch;
###############################################################################
# Variables

###############################################################################
# Nagios file handlers
my $host_file = "/usr/local/nagios/etc/objects/hosts.cfg";
my $group_file = "/usr/local/nagios/etc/objects/hostgroups.cfg";
my $services_file = "/usr/local/nagios/etc/objects/services.cfg";
my $unixmatch = "sdh-unix" ;
my $windowsmatch  = "sdh-windows";
my $vpsmatch = "vps-server";
my $date = `date "+%d%m%y-%H%M%S"`;

###############################################################################
# Verify Arguments
if ((!$ARGV[0]) || (!$ARGV[1])) {
        &usage;
}

if (length($ARGV[1]) gt 1 ) {   print "Command options too long!\n"; &usage; }

# Verify Nagios is working before we start
my $nagios = `nagios -v /usr/local/nagios/etc/nagios.cfg`;
        if ($nagios =~ /One or more problems was encountered while processing the config files/) {
        print "CRITICAL ERROR!\n\nNagios is already broken and we cannot continue!\nPlease fix it!\n";
                @error_array = split(/\./, $nagios);
                for $error (@error_array) {
                $error=~s/^\n//g;
                        print "$error\n" if  $error=~ /Error:/;
             }
        die "\n\nProgram Aborting before even starting due to nagios config error!\n"

}

# Clean up any old tmp files.
unlink("/tmp/hosts.cfg.tmp");
unlink("/tmp/hostgroups.cfg.tmp");
unlink("/tmp/services.cfg.tmp");

###############################################################################
# The Main Program control statement.
###############################################################################
switch ($ARGV[1]) {

        case /d/i {     &delete;        }
        case /x/i {     &addEntry("x"); }
        case /w/i {     &addEntry("w"); }
        case /v/i {     &addEntry("v"); }
        else {
                print "Option: $ARGV[1] not found \n";
                &usage;
        }
}
###############################################################################

###############################################################################
# Subroutines
###############################################################################

###############################################################################
## sub backup - Backs up the nagios config files that are to be modified
###############################################################################
sub backup {

# Backup The Nagios files into an archive.
        $date =~s/\n//g;

        mkdir("/var/backup/nagios/$date", 0755 ) || die "Cannot create directory /var/backup/nagios/$date\n";

        copy($host_file, "/var/backup/nagios/$date/hosts.cfg.bck"); #|| die "Cannot copy $host_file to /var/backup/nagios/$date/hosts.cfg.bck\n";
        copy($group_file, "/var/backup/nagios/$date/hostgroups.cfg.bck"); #|| die "Cannot copy $host_file to /var/backup/nagios/$date/hostgroups.cfg.bck\n";
        copy($services_file, "/var/backup/nagios/$date/services.cfg.bck"); #|| die "Cannot copy $service_file to /var/backup/nagios/$date/services.cfg.bck\n";
}

###############################################################################
## sub openFile($filename) - returns the file to a buffer for parsing
###############################################################################
sub openFile {
        my $blob;
        my $file = shift;
        open (F, "< $file") or die "Can't open $file : $!";

        while(  ) {
                $blob .= $_;
        }

        close(F);
return $blob;

}

###############################################################################
###############################################################################

###############################################################################
## sub delete - Deletes the servername from the config files.
###############################################################################
sub delete {
        &backup;                # Backup the files before we do anything to them.
        &delete_host;
        &delete_hostgroup;
        &delete_services;
        &checkNagios;
}

###############################################################################
## sub delete_host - deletes the host entry from hosts.cfg
###############################################################################
sub delete_host {
        my $host_str = &openFile($host_file);
        my $pattern=$ARGV[0];   # The parser doesn't like the array so we just pass it to a variable.

        # parse the hosts.cfg file first
        # This regular expression is a defined host entry, if it can't find it
        # and assert that the hostname is part of that context, it will die.
        if ($host_str =~/define[^_]*.name.*(?s-i:$pattern)[^}]*./i) {
                print "command: $ARGV[1] : Deleting $ARGV[0] $1\n" if $host_str =~s/define[^_]*.name.*(?s-i:$pattern)[^}]*.//g;
                print "Match: $ARGV[0]\n" if $host_str =~/define[^_]*.name.*(?s-i:$pattern)[^}]*./i;
                print "Deleted $ARGV[0] from hosts.cfg\n";

        # Write the successfull deleteion to a tmp file.
        open(HF, ">/tmp/hosts.cfg.tmp") || die "Cannot open /tmp/hosts.cfg.tmp";
        print HF $host_str;
    close(HF);

        } else { die "Could not find and entry for $ARGV[0] in hosts.cfg\n"; };
}

###############################################################################
## sub delete_hostgroup - deletes the hostgroup entry
###############################################################################
sub delete_hostgroup {

        my $hostgrp_str = &openFile($group_file);
        my $pattern=$ARGV[0];   # The parser doesn't like the array so we just pass it to a variable.

        # search/replace the hostgroup.cfg file
        if ($hostgrp_str =~ /$pattern/i) {
                # If the server has a comma after it, we need to remove that too.. or breakage.
                if ($hostgrp_str =~ /$pattern,/i ) {
                        print "Deleted $ARGV[0], from hostgroups.cfg\n" if $hostgrp_str =~ s/$pattern,//g;
                }

                        print "Deleted $ARGV[0] from hostgroups.cfg\n" if $hostgrp_str =~ s/$pattern//g;

        } else {
                die "Could not find and entry for $ARGV[0] in hostgroups.cfg\n";
        }
        open(HGF, ">/tmp/hostgroups.cfg.tmp") || die "Cannot open /tmp/hostgroups.cfg.tmp";
        print HGF $hostgrp_str;
        close(HGF);
}

###############################################################################
## sub delete_services - delete the serivices.cfg entry
###############################################################################
sub delete_services {

        my $services_str= &openFile($services_file);
        my $pattern=$ARGV[0];   # The parser doesn't like the array so we just pass it to a variable.

        # search/replace the hostgroup.cfg file
        if ($services_str =~ /$pattern/i) {
                # If the server has a comma after it, we need to remove that too.. or breakage.
                if ($services_str =~ /$pattern,/i ) {
                        print "Deleted $ARGV[0], from services.cfg\n" if $services_str =~ s/$pattern,//g;
                }
                        print "Deleted $ARGV[0] from services.cfg\n" if $services_str =~ s/$pattern//g;
         } else {
                die "Could not find and entry for $ARGV[0] in services.cfg\n";
        }
        open(SF, ">/tmp/services.cfg.tmp") || die "Cannot open /tmp/services.cfg.tmp";
        print SF $services_str;
        close(SF);
}

###############################################################################
## sub checkNagios - checks nagios for errors and rolesback if so.
###############################################################################
sub checkNagios {

        copy("/tmp/hosts.cfg.tmp", $host_file) || print "Cannot copy /tmp/hosts.cfg.tmp to  $host_file\n";
        copy("/tmp/hostgroups.cfg.tmp", $group_file) || print "Cannot copy /tmp/hostgroups.cfg.tmp to $host_file\n";
        copy("/tmp/services.cfg.tmp", $services_file) || print "Cannot copy /tmp/services.cfg.tmp $service_file\n";
        my $success = `nagios -v /etc/nagios/nagios.cfg`;

        if ($success =~ /One or more problems was encountered while processing the config files/) {
                print "CRITICAL FAILURE - See Errors!\n";
                @error_array = split(/\./, $success);
                for $error (@error_array) {
                $error=~s/^\n//g;
                        print "$error\n" if  $error=~ /Error:/;
                        }
                print "\nRestoring from backup\nCheck /tmp/hosts.cfg /tmp/hostgroup.cfg /tmp/service.cfg\n";
        copy("/var/backup/nagios/$date/hosts.cfg.bck", $host_file) || die "Cannot copy /var/backup/nagios/$date/hosts.cfg.bck to $host_file\n";
        copy("/var/backup/nagios/$date/hostgroups.cfg.bck", $group_file) || die "Cannot copy /var/backup/nagios/$date/hostgroups.cfg.bck to $group_file\n";
        copy("/var/backup/nagios/$date/services.cfg.bck", $services_file) || die "Cannot copy /var/backup/nagios/$date/services.cfg.bck $services_file\n";

        } else {
                print "Nagios config reports success, restarting nagios\n";
                my $restart = `/etc/init.d/nagios reload`;
                print $restart;
        }

}

###############################################################################
## sub addEntry - adds the unix or windows host entry.
###############################################################################
sub addEntry {

        my $type = shift;
        my $pattern = $ARGV[0];
        my $host_str = &openFile($host_file);
        my $hostgrp_str = &openFile($group_file);
        my $services_str= &openFile($services_file);

        if ($host_str=~/$pattern/) { die "$ARGV[0] already in hosts.cfg, aborting!\n"; }
        if ($hostgrp_str=~/$pattern/) { die "$ARGV[0] already in hostgroups.cfg, aborting!\n"; }
if(($type eq 'w') || ($type eq 'x')) {
        if ($services_str=~/$pattern/) { die "$ARGV[0] already in services.cfg, aborting!\n"; }
}
        # Some sanity checks to help prevent data entry errors
        if (!$ARGV[2]) { print "\nNo Server Alias, aborting!\n\n"; &usage; }
        #if ($ARGV[2]=~/[0-9]{5,8}$/i) {} else { print "No Member ID!\n"; exit 0}
        if (!$ARGV[3]) { print "\nNo IP Address specified, aborting!\n\n"; &usage; }
        if ($ARGV[3]=~/[a-z]/i) { print "\nIP Address $ARGV[3] is invalid, please double check\n\n"; exit 0; }
        if ($ARGV[3]=~/[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/) {
                @ip = split(/\./, $ARGV[3]);
        } else {
                print "\nIP Address $ARGV[3] is invalid, please double check\n\n"; exit 0;
        }

        if (($ip[0] > 254) || ($ip[1] > 254) ||
            ($ip[3] > 254) || ($ip[4] > 254)) {

        print "\nIP Address $ARGV[3] is invalid, please double check\n\n"; exit 0;
        }

# Passes sanity checks, back up the mo fo.
&backup;
# Check if windows or unix

switch ($type) {

        case "x" {      print "Unix!\n";
                        $hostgrp_str =~ s/$unixmatch/$unixmatch,$pattern/g;
                        $services_str =~ s/$unixmatch/$unixmatch,$pattern/g;  }
        case "w" {      print "Windows\n";
                        $hostgrp_str =~ s/$windowsmatch/$windowsmatch,$pattern/g;
                        $services_str =~ s/$windowsmatch/$windowsmatch,$pattern/g; }

        else {          print "VPS\n";
                        $hostgrp_str =~ s/$vpsmatch/$vpsmatch,$pattern/g;
                                }
        } # end switch

        # Add it to the host_str buffer.
        $host_str .= "define host{
        use                     sdh-dedicated
        host_name               $ARGV[0]
        alias                   $ARGV[2]
        address                 $ARGV[3]
        }\n\n";
        open(HF, ">/tmp/hosts.cfg.tmp") || die "Cannot open /tmp/hosts.cfg.tmp";
        print HF $host_str;
        close(HF);
        open(HGF, ">/tmp/hostgroups.cfg.tmp") || die "Cannot open /tmp/hostgroups.cfg.tmp";
        print HGF $hostgrp_str;
        close(HGF);

if(($type eq 'w') || ($type eq 'x')) {
        open(SF, ">/tmp/services.cfg.tmp") || die "Cannot open /tmp/services.cfg.tmp";
        print SF $services_str;
        close(SF);
        }
        &checkNagios;

}
###############################################################################
## sub usage - prints the usage when things don't add up from args
###############################################################################
sub usage{
        print "Usage: /usr/local/bin/nagios-add.pl    \n\n";
        print "Optional Flags:\n";
        print "\td delete a server\n";
        print "\tw add a windows server\n";
        print "\tx add a unix server\n";
        print "\tv add a VPS server\n\n";
        print "eg delete:\t./usr/local/bin/nagios-add.pl sdh-server12 d\n";
        print "eg add:\t\t./usr/local/bin/nagios-add.pl sdh-server12 x \"sdh-server12 sdh-server12.stardothosting.com MemID:155\" 192.168.111.10\n";
        exit 0;
}

1 Comment

MySQL Replication : Replicating an existing database

You may remember a previous post about MySQL replication.

I decided to make a revised post detailing the different steps required in order to implement a master / slave replication relationship within two or more MySQL servers.

The steps required are slightly different and I think its important to outline the necessary steps in order to accomplish this task — it may actually save you some troubleshooting! :)

    Replication of Existing DBs

If you have existing data on your master that you want to synchronize on your slaves before starting the replication process, then you must stop processing statements on the master, obtain the current position, and then dump the data, before allowing the master to continue executing statements.

If you do not stop the execution of statements, the data dump and the master status information that you use will not match and you will end up with inconsistent or corrupted databases on the slaves.

    PREPARATION OF MASTER SERVER

1. Select a master server. It can be either one.

2. Make sure all databases that you want to replicate to the slave already exist! The easist way is to just copy the database dirs inside your MySQL data directory intact over to your slave, and then recursively chown them to “mysql:mysql”. Remember, the binary structures are file-system dependant, so you can’t do this between MySQL servers on different OS’s. In this instance you will want to use mysqldump most likely.

3. Create /etc/my.cnf if you do not already have one:

[mysqld]
socket=/tmp/mysql.sock [enter YOUR path to mysql.sock here]
server-id=1
log-bin=mysql-bin
binlog-do-db=bossdb     # input the database which should be replicated
binlog-ignore-db=mysql1 # input the database that should be ignored for replication
binlog-ignore-db=mysql2  # input the database that should be ignored for replication

4. Permit your slave server to replicate by issuing the following SQL command (substituting your slave’s IP and preferred password):

mysql> GRANT SUPER,REPLICATION CLIENT,REPLICATION SLAVE,RELOAD ON *.* TO 'replicate'@'192.168.1.1' IDENTIFIED BY 'somepass';

5. Flush all talbes and block write statements :

mysql> FLUSH TABLES WITH READ LOCK;

6. Use the SHOW MASTER STATUS statement to determine the current binary log file name and offset on the master:

mysql > SHOW MASTER STATUS;
+---------------+----------+--------------+------------------+
| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+---------------+----------+--------------+------------------+
| mysql-bin.003 | 73       | test         | manual,mysql     |
+---------------+----------+--------------+------------------+

Copy the file + position for use in Step 4 of the slave configuration.

7. Create data snapshot to import into slave with mysqldump :

shell> mysqldump -u root -p db_name --lock-all-tables >dbdump.sql

8. Unlock the tables of the database :

mysql> UNLOCK TABLES;

9. Transfer & import the db into the slave

10. Shut down and restart MySQL daemon and verify that all is functional.

PREPARATION OF SLAVE

1. Create /etc/my.cnf if you do not already have one:

[mysqld]
socket=/tmp/mysql.sock [enter YOUR path to mysql.sock here]
server-id=2 [MUST be different to master]
master-host=192.168.1.1
master-user=replicate
master-password=somepass

2. Shut down and restart MySQL on slave.

3. Log into mysql and stop slave :

mysql> stop slave;

4. Set the master configuration on the slave :

mysql> CHANGE MASTER TO
    ->     MASTER_HOST='master_host_name',
    ->     MASTER_USER='replication_user_name',
    ->     MASTER_PASSWORD='replication_password',
    ->     MASTER_LOG_FILE='recorded_log_file_name',
    ->     MASTER_LOG_POS=recorded_log_position;

3. Issue the following SQL command to check status:

mysql> show slave status\G;

Ensure that the following two fields are showing this :

Slave_IO_Running: Yes
Slave_SQL_Running: Yes

If not, try to issue the following command :

mysql> start slave;

This will manually start the slave process. Note that only updated tables and entries after the slave process has started will be sent from the master to the slave — it is not a differential replication.

TESTING

Just update some data on the master, and query that record on the slave. The update should be instantaneous.

Test creating a table on the master MySQL server database :

mysql> use replicateddb;
Database changed

mysql> CREATE TABLE example4( id INT NOT NULL AUTO_INCREMENT,  PRIMARY KEY(id),  name VARCHAR(30),   age INT);
Query OK, 0 rows affected (0.04 sec)

2 Comments

Shell Script to Report On Hacking Attempts

It is always a good idea , when implementing open source firewall implementations (iptables, pf, etc), to build in as much reporting and verbosity as possible.

Having verbose reports on the state of your firewall, intrusion attempts and other information is key to ensuring the health and integrity of your network.

Somewhere along the line, we wrote a script to provide daily reports on intrusion attempts to penetrate our network — this usually happens when someone exceeds certain connection thresholds.

It may not be the most informative data, but the script can be modified to provide other important statistical information. It can also be modified to be used with other firewall implementations. I’m certain it wouldn’t be hard to convert this script to utilise iptables.

Below you will find the script itself — it can be set to run daily as a cronjob perhaps. Also note that the script tries to resolve a hostname for the IP address to at least provide some quick & easy information to the security administrators when determining coordinated attacks or attacks coming from compromised systems.

#!/bin/bash
# SDH PFCTL Daily Hack Table check

yesterday1=`date -v -1d +"%b"`
yesterday2=`date -v -1d +"%e"`
yesterday_display=`date -v -1d +"%b %d %Y"`

echo "" > /var/log/tablecheck.log

/sbin/pfctl -vvsTables > /var/log/pfctltables.log

echo "Firewall Table Audit: " $yesterday_display >> /var/log/tablecheck.log
echo -e "----------------------------------">> /var/log/tablecheck.log
echo -e "" >> /var/log/tablecheck.log

for obj0 in $(cat /var/log/pfctltables.log | grep "\-pa\-r\-" | awk -F "\t" '{printf "%s\n", $2}');
do
echo -e $obj0 "TABLE" >> /var/log/tablecheck.log
echo -e "--------------" >> /var/log/tablecheck.log

# this is because the date command outputs single digit non-aligned right, but pfctl doesnt display that way :(
if [ "$yesterday2" -le 9 ]
then
        /sbin/pfctl -t $obj0 -Tshow -vv | grep -A 4 -B 1 "$yesterday1  $yesterday2" >> /var/log/tablecheck.log 2>&1
else
        /sbin/pfctl -t $obj0 -Tshow -vv | grep -A 4 -B 1 "$yesterday1 $yesterday2" >> /var/log/tablecheck.log 2>&1
fi

if [ "$?" -eq 1 ]
then
        echo -e "No values found for yesterday" >> /var/log/tablecheck.log
        echo -e "" >> /var/log/tablecheck.log
else
        echo -e "Hostnames :" >> /var/log/tablecheck.log
        for obj1 in $(/sbin/pfctl -t $obj0 -Tshow -vv | grep -B 1 "$yesterday1 $yesterday2" | grep -v "Cleared" | grep -v "\-\-");
        do
        iphostnm=`/usr/bin/nslookup $obj1 | grep -A1 "Non-authoritative answer" | grep "name" | awk -F "=" '{printf "%s\n", $2}'`
        if [ "$?" -eq 0 ]
        then
                echo -e "$obj1 / $iphostnm" >> /var/log/tablecheck.log
        else
                echo -e "$obj1 / No host name found" >> /var/log/tablecheck.log
        fi
        done
       echo -e "" >> /var/log/tablecheck.log
fi

done

cat /var/log/tablecheck.log | mail -s "Firewall Table Report" you@youremail.com

Enjoy!

1 Comment