I’ve commented now and again that I’d be willing to pay upwards of $100 for a handheld version of Wikipedia — one that I can sync via WiFi, then take on the bus and read to my heart’s content. Unfortunately, the market doesn’t seem to be responding to my lone cry… so I wrote a Perl script that loads Wikipedia onto my iPod.

Wikipedia on the iPod

Wikipod starts from a Wikipedia page and traverses all the links it finds, downloading each page to the iPod. It will then traverse each link on the new pages, continuing this process until it finds no new links or has reached the size limit you specify.

To get started, download the script and mark it as executable:

1. Download wikipod to your desktop and unzip it
2. Open Terminal.app and type cd ~/Desktop
3. Type chmod 755 wikipod to mark the script as executable

Then, each time you want to load Wikipedia pages onto your iPod, do the following:

1. Mount your iPod in disk mode (its icon should appear on the Desktop)
2. Open Terminal.app and type cd ~/Desktop
3. Pick a Wikipedia page to start from. We’ll choose the iPod page for this example
4. Type ./wikipod http://en.wikipedia.org/wiki/IPod, then a space
5. Double-click on your iPod icon and drag the Notes folder into the Terminal window
6. Type the number of megabytes of space for wikipod to fill

The command should now look like this:

./wikipod http://en.wikipedia.org/wiki/IPod /Volumes/iPod/Notes 10

Hit enter — and your iPod will start filling up with Wikipedia entries! Go back to your iPod folder, double-click on the Notes folder, and watch the entries as they’re downloaded, formatted for your iPod, and transferred over. When the script exits and prints “Done transferring n pages!”, you can unmount your iPod.

There’s one caveat about using notes on the iPod — the first time you navigate to the Notes menu after syncing with iTunes or downloading new Wikipedia articles, the iPod needs to inspect each page to see whether the links are valid. This can take a fair amount of time (and battery, since it’s disk-intensive), so I recommend leaving the iPod plugged in until it finishes. (This is a real bummer — I’m looking into ways to work around it, but I’m not optimistic.)

Modifying this script to be a general-purpose web-page-to-iPod-note translator is left as an exercise for the reader…

Update: I’ve fixed several of the problems noted in the comments below — namely, slashes in the URL no longer break the script, users can leave off the trailing slash in the directory path, and I’ve put a 50MB limit on how much can be downloaded at one time.

I wholeheartedly appreciate the tips and suggestions that have been offered — thanks!

Update 2: I added support for UTF-8 characters (Greek, for example) per this technote from Apple.

Update 3: Apple limits the iPod to 1000 notes, so my script stops after that many. This will be revised if the iPod limitation is lifted.

Update 4: The script now sleeps for two four seconds between each page request, to comply with the Wikipedia robots.txt file. While Wikipedia discourages crawling their site, these limits should prevent wikipod from causing undue harm. Users who run into problems using wikipod are encouraged to email me or leave a comment on this post.

Update 5: Be sure to check out these instructions for using Automator to accomplish the same thing.

Update 6: A brief article on constructive comments.

Update 7: David Still improved wikipod by adding user feedback showing how many iPod notes have been transferred. He also wrote a great review of the script. Nice work, David!