Perl advent post Dec. 24th

Over the Christmas holiday I wrote a few posts for the Perl advent calendar. I'd like to share them here, if only for posterity. I believe you should be able to view the archived original.

Perl Advent Calendar 2006-12-24
Santa's Dilemna
by Ben Prew

Kris Kringle had just left the North Pole on his brand-new state-of-the-art sleigh, complete with wireless access to the North Pole presents database. The elves had been hard at work bringing present distribution into the 21st century, and their finishing touch was getting rid of Santa's old, outdated hand-written list. The who-is-naughty, who-is-nice, and who-wants-what lists were now stored in a fancy new Oracle(!) database. (Yes, Oracle got off the naughty list with that one)

Everything was going swimmingly and Kris was picking, packing and delivering more presents then ever (Which is a good thing, considering our "conversion" program currently going on in Iraq). Unfortunately, just as he was sweeping into Russia, a black-bellied plover slammed into the side of the sleigh! Kris looked over the side and to his horror, a smoking hole was all that remained of his wireless link to the North Pole. A crackling on the radio brought Kris back to his senses. It was Rudolph, his main reindeer.

"Captain, what was that?"

"Rudy, we just got groused and lost our NP link."

"KGB?"

"Uhh… I don't think so."

Rudolph was a huge conspiracy theory freak, and of course since they were over Russia, it could only be the KGB trying to sabotage Kris. They were trying to enact a "regime change", at least according to the reindeer.

Kris began rooting through his system, looking for a second chance… anything….

Any sort of land line connection… no
Cell-phone card… not even Verizon has that much coverage….
Tin-foil to make an improvised antenna from the reindeers antlers…. no

Wait, wait… maybe this was something….

Kris realized that Norrish, his head elf, had been using his laptop to munge the wish list before importing it into the new database, and the raw text files were still sitting on his hard drive! Unfortunately, Kris didn't have time to install Oracle, much less the horsepower to run the database and import the data. He slumped back in his seat, slowly coming to terms with what this meant. Just as he was about to turn the sleigh around, Rudolph's nose lit up…

"What is it boy, what are you thinking?"

Rudolph started wagging his tail excitedly (yes, reindeer have tails), and began jostling up and down, braying and making other reindeer noises.

"Come on Rudy, you know I can't understand you when you're this excited. Slow down, take a deep breath and try again"

Rudolph inhaled deeply, exhaled, and then quitely, calmly uttered four meager syllables,

"Any data?"

Kris stared at Rudolph blankly, not quite sure what to make of what was just said. What in the blazes did that mean? Out of ideas, but with little else to do over the vast Siberian wasteland, Kris focused all his Kringle energies and attempted to decode his companion's cryptic quip.

He scanned his memory, trying to dig up something, anything at all. Something nagged at the back of his mind, he could feel it, but he just couldn't mold it into a coherent thought. Suddenly, his eyes lit up, and quick as a wink Kris knew what Rudolph meant. The code began pouring from his fingers onto the screen as Kris entered the zone. His eyes narrowed as all other thoughts were put aside, as he intently focused on the task at hand.

"Where were those files… in Norrish's home directory of course"

"Now, the sql queries"

With furious typing, a little elf magic and much unit testing (You know he rolls with XP), Kris had solved his problems and saved Christmas. Before you scroll down to the code, can you guess what characters his keyboard had expressed? What masterpiece he was able to piece together in a few short minutes, with only raw text files, and but a word from his most trusted reindeer? Behold, the code of Kris Kringle!
mod24.pl - Kringle's Code

1 #!/usr/bin/perl
2
3 use DBI;
4
5 # My files are laid out like this:
6 #
7 # presents.txt
8 # person_no|present
9 # 1|bicycle
10 # 1|action figure
11 # 2|doll
12 # 3|doll
13 # 4|bicycle
14 #
15 # people.txt
16 # person_no|name|personality|address
17 # 1|bob smith|nice|123 anywhere st, St. Petersburg
18 # 2|alice andrews|nice|465 somewhere st, Moscow
19 # 3|frank martonick|naughty|1138 lenin ave, St. Petersburg
20 # 4|billy cutter|nice|31337 peoples lane, Moscow
21
22 my $dbh = DBI->connect('dbi:AnyData(RaiseError=>1):');
23
24 $dbh->func( 'people', 'Pipe', '/home/norrish/lists/people.txt', 'ad_import');
25 $dbh->func( 'presents', 'Pipe', '/home/norrish/lists/presents.txt', 'ad_import');
26
27 my $sth = $dbh->prepare(q/
28 SELECT person_no, name, address
29 FROM people
30 WHERE personality = ?/);
31
32 $sth->execute('nice');
33
34 while ( my $person = $sth->fetchrow_arrayref ) {
35 my $presents = $dbh->selectall_arrayref(q/
36 SELECT present
37 FROM presents
38 WHERE person_no = ?/, {}, $person->[0]);
39
40 print "Presents for ". $person->[1] . " (living at: ". $person->[2] . ")\n\t",
41 join("\n\t", map { join " ", @$_ } @$presents), "\n\n";
42 }
43
44 $dbh->disconnect();

DBD::AnyData is a Perl module that allows one to, among other things, use a variety of (mostly text) file formats as tables in a database. It seems to have limited support for joins, so I had to do the queries separately. Even so, AnyData is really cool!

Perl advent post Dec. 17th

Over the Christmas holiday I wrote a few posts for the Perl advent calendar. I'd like to share them here, if only for posterity. I believe you should be able to view the archived original.


Perl Advent Calendar 2006-12-17
Yule Log-Rolling
by Ben Prew

When running automated processes, I find it incredibly useful to have some sort of logging setup, so that I can see how long certain parts of processing take. Or, even more importantly, if the process dies, I can better determine what it was doing shortly before it bit the dust.

At work, we have many automated and semi-automated processes that run at a scheduled time. These processes all log to the same directory, which makes it easier to find them. Also, I would like to automatically rotate new files when they show up in this directory, and not have to deal with any sort of configuration file.

I could have done this with logrotate, or some other process, but I like doing things in Perl, and I didn't want to interfere with existing archiving processes on the box. With Logfile::Rotate I can eat my bûche de Nöel and have it too.

If I wanted to write a separate script to rotate all the log files for me, it might look something like mod17e.pl (external):

1 #!/usr/bin/perl;
2
3 use Logfile::Rotate;
4
5 my @logs = map {
6 my $file = $_;
7 Logfile::Rotate->new(
8 File => $file,
9 Gzip => 'lib',
10 Dir => '/var/logs/dev.old',
11 Post => sub { unlink $file } ); } ;
12
13 for (@logs) { $_->rotate() }

The default behavior is to leave an empty log file in the directory, but all my processes will create their own files, if needed, so I would rather just remove the file. This was easy to add with the Post argument.

Another benefit of Logfile::Rotate is that unlike an external binary, I can embed it in my existing code. All of our current logging is done though a mix-in, so I've got a single point of contact for each process that runs, regardless of where it logs to.

This method is called log(), and it handles all the logging for each file, as well as knowing which file to log to. This also gives me more flexibility in how each log is rotated. A rotation could be triggered by the process catching a signal, the number of logged messages exceeding some threshold, the time elapsed since last rotation, or the log growing too large (in an effort to avoid filling the partition), etc.

So, if I wanted to rotate each log file at 100 Mb, regardless of when it was last rotated, the code might look something like mod17i.pl (internal):

1 sub log
2 {
3 my ($self, $message) = @_;
4
5 # logging stuff here.
6
7 if ( ( -s $self->filename ) > 100_000_000) {
8 Logfile::Rotate->new(
9 File => $self->file_name,
10 Gzip => 'lib',
11 Dir => '/var/logs/dev.old',
12 )->rotate;
13 }
14 }

Having the log files rotate themselves, how great is that! Now we don't have any other external scripts or configurations to maintain. Of course, the downside to this approach is the implicit stat() on each call to log(), but it shouldn't add too much overhead. This can even be alleviated if there is only one process that writes to the log file, since you could then have a counter that is initialized to the current size of the file and then adds the size of the message to the counter. Then, once the counter reached 100_000_000, you could rotate the log file.
SEE ALSO
Log::Dispatch::FileRotate, logrotate(8)