I've been envying a guy at work who lugs his PowerBook G4 17" into the office everyday--the panther GUI really does have amazing style and polish. I take the scenic route, passing his cube, to the coffee two or three times a day; sometimes, when he's not around, I admit I walk up and rub it softly and promise to come back for her when I have an extra $3,000 laying around.

Milk is pretty heavy on the pixmaps. I tried it for about 10 minutes about a week ago and was just completely disappointed at how much it slowed things down. Not now. I'm back around the same responsiveness I got with Bluecurve under GTK+ 2.2, which wasn't bad by a long shot. It's just kind of amazing how you can get serious system-wide increases by optimizing in sweet spots. We should be looking for more of these.

Back to the point of this entry... URLGrabber is really starting to come together and we hope to announce an early test release in the near future. Michael set up a nice project page on Duke's Linux site that has a significant amount of information, viewcvs, and all the other goodies.

Here's the list of features ripped from the project page for your convenience.

• identical behavior for http://, ftp://, and file:// urls
• http keepalive - faster downloads of many files by using only a single connection
• byte ranges - fetch only a portion of the file
• reget - for a urlgrab, resume a partial download
• progress meters - the ability to report download progress automatically, even when using urlopen!
• throttling - restrict bandwidth usage
• batched downloads using threads - download multiple files simultaneously (feature still in progress)
• retries - automatically retry a download if it fails. The number of retries and failure types are configurable.
• authenticated server access for http and ftp
• proxy support - support for authenticated http and ftp proxies
• mirror groups - treat a list of mirrors as a single source, automatically switching mirrors if there is a failure

Not to shabby. I plan on blogging a bit about some of the cooler things you can do with urlgrabber once we have a clean release out there.

Proof

Stoked. :)

I've heard many arguments for why Intellectual Property is bad. Most have the word "freedom" or "unfair" in them. In general most anti IP arguments are from the perspective of how big business is infringing on the rights of individuals or this great country or whatever. But here's an argument that you probably haven't heard that is a bit different: IP is expensive for businesses and the cost is rising steadily. I don't mean that it costs a lot to register a patent or copyright, or even that it costs a lot to litigate when IP is infringed upon. I am sure all that is expensive and you might make an argument that it is not worth it based solely on those aspects. However, those costs might be nothing compared to the amount of overhead businesses incur to protect IP at the operational and security levels.

I'm trying to get people at work to publish information on the projects they're working on; any information, all information. Formal documentation is great but we need more than that. We want informal stuff. The type of stuff you see in email discussions [mailing lists, forums] and journals of what people and teams are doing [weblogs]. I want that information available in some text based format (HTML, XML, whatever I don't care) and I want it all on the web—not public, of course, but it needs to be web enabled [HTTP] on the LAN. I want feeds for syndication [RSS, Atom] and everything aggregated somewhere [Schwag w00t!]. This must all be searchable [htdig]. At the end of the day I want anyone/anywhere in the company to be able get information on what anyone/anywhere in the company is doing, right now. When someone finds out what someone is doing and they think it is useful they need to be able to keep track of what they're doing. This is all happening now in the FS/OSS world and it works, it can work for us too. It could all be so beautiful, it could save the company tons of money, you would have to ship the money to the bank in dump trucks, we could build another building just for the money and swim in it whenever we wanted, I can smell the money it is so close... But wait! Recent company policy has mandated that NO information revealing details of IP owned by company or portions thereof be publicly available on the company Intranet. All information must be password protected and access to said information must not be granted to any individual without approval by [insert some important person who knows who is and is not to be trusted here].

Okay, so all of this hasn't really happened, yet. But I am being asked to require authorization for certain pieces of information and to otherwise limit availability of information. The reasons seem valid, I guess. It seems that some place at some time, some company hired in some consultant who smuggled out some IP related information and then sold it to some competitor. One of the funnier reasons is that there are apparently some employees who do nothing but print stuff they find on the LAN and then immediately throw it away—it is available later that night for the dumpster divers. And then there is the worry of someone hacking into the LAN and having all this information available at there finger tips. All valid concerns when protecting IP.

But I'm not arguing against the idea that you need strong security if you want to protect all of this IP, I am arguing whether the expense of protecting all of this IP is worth the value of the IP. I think I have a pretty good vantage point for evaluating the costs—at least with regards to efficiencies. I work at a large corporation by day (which I love by the way; very good company) and do Free Software development at night and on weekends. From here, one of the first things I observe is that the FS/OSS community has a huge asset in that they do not have to protect IP (at least, not the way businesses do). Having everything open means all of the benefits I outlined previously. Further, the value of this asset will increase as the amount of information increases and types of information widen. When I look over to the proprietary model, this massive growing asset (information) somehow becomes a liability with huge expense.

So how much is the IP really worth? I'm not asking rhetorically, I feel I've given a glimpse into the costs from the operational side now I want to know, how much is this stuff worth?

Me: Need a global variable here.
Little Angel: No you don't. Globals are always bad!
Little Devil: Shaddap.. slap, slap. We just need to do this thing real quick. This isn't java.
Me: Hmm.. Maybe we can just use automatic scoping..

>>> x = 5
>>> def y():
...   print x
...
>>> y()
5

Me: Good. So I just use it then.. Hmm.. but wait, surely there will be name clashes..

>>> x = 5
>>> def y():
...   x = 10
...
>>> print x
5
>>> y()
>>> print x
5
>>> # aha!
>>> def y():
>>>    global.x = 10
  File "<stdin>", line 2
    global.x = 10
          ^
SyntaxError: invalid syntax
>>>    __main__.x = 10
>>> y()
>>> print x
5
>>> # ughh
>>> def y():
>>>   ../x # yea right.
[Ctl-D,Ctl-D]

Me: Sneaky little thing you.. So, you're automatic unless I scope something in local. I'm going to need some help here..

Google: python globals
Me: "18ft snake eats glowing balls of..", huh?

Google: python global variable scope
Me: Better. Here's a good code snip:

x = 5
def somefunc():
   global x
   print x

Me: ahh..

>>> x = 5
>>> def y():
...    global x
...    x = 10
...
>>> y()
>>> print x
10

I now know how to use globals in python. That looks like a lot of work but that probably took all of 2 minutes. I guess I could have bypassed the initial tests and went to google first but I was able to find out a lot with simple trial and error.

I'm lead to the opinion that there are two kinds of programmers and two kinds of languages. You have the scientific, trial and error, show-me-the-source type hacker and then you have the businessy, formal documentation, show-me-the-sdk type. If I had to slot languages I would say C, Python, Perl, and maybe Lisp all seem to fall into the first category and Java, C++, C# fall into second. Some languages, like the different BASICs, seem to fall somewhere in between.

It seems that one of the key benefits of the "learn as you go" languages, and Python specifically, is that you can apply much of what you've learned in other languages very quickly. I have a decent background in Java, C, and Perl. I'm constantly amazed at how much I already know in python by simply asking myself questions of the form, "If I had to write a programming language that did XXX from YYY, how would I write it?" For example, "If I had to write PACKAGE MANAGEMENT from JAVA, how would I write it?" It would look a lot like Python. "If I had to write LIST and HASH stuff from PERL, how would I write it?" Again, it would look a lot like python (well, it would be a lot less useful, I'm sure).

Python really is a language that you can learn as you go, especially if you have experience in other languages. I'm constantly surprised by how much of the language I already know but don't remember learning. Combine that with the ability to do rapid and productive trial and error'ing and you have a language that is sure to be a hit with the scientific types, as is already clearly established.

My basic play style thus far has been as follows. I use the FG42 as a rifle. I haven't seen anyone say one good thing about this gun but I seem to win more than not in SMG battles. I rarely snipe and pretty much act like a soldier/rambo, except I have satchel charges and can wear enemy uniforms. Speaking of satchel charges, they rock. The best weapon in the game, IMO. I try to always have a satchel sitting at a frequented passageway and then run around defending myself with my rifle until I can blow the satchel. This means you have to "have your head on a swivle", watching for guys to shoot but constantly glancing at the satchel for a kill their. Besides kills, my big goal in the game is to blow up as many enemy structures as possible (outposts, turrets, bridges, etc). Lastly, one thing I've found extremely useful in long (10 map+) campaigns is to start as a Medic and work your ass off for the Adrenaline shot. Once you get the Adrenaline you can switch to Covert Ops and still use the Adrenaline. This pretty much makes you unstoppable in my experience. I figured out that last bit on accident after switching to Covert Ops initially.

So tonight I've been on a quest for other people's Covert Ops strategy/tactics. I figured I would linkdump what I found useful.

RTCW & ET 4 Newbies - Covert Ops
Good overview of Cov. Ops /w pictures. Nothing to spectactular regarding more advanced strategies.

http://wolfensteinresource.com/forum/board21/thread1100.htm
Wolfenstein Resource forum post that goes into good depth on Cov. Ops tactics. A little more orthodox than how I've been playing though.

GameFAQs : ET : Covert Ops Guide
In progress at time of writing. This has detail on how to level up and what you get for leveling in different areas. Should be good when finished.

Here's some video (MOV).

I'm just kind of frustrated. It may be that what I'm looking for is a lot of different stuff that needs broken up into separate projects. For instance, schwag could provide the aggregation and feed normalization and another app could handle the whole bookmark management stuff. It should be trivial to integrate normalized feeds into the bookmark manager.

Seth teased me with suggestions of trying to get some of the urlgrabber/byterange stuff into python core when we started. Running back through the code had me thinking that a lot of the FTP byterange stuff would fit better as a urllib(2) patch anyway.

Along with normalized Atom feed generation, the feeder component dumps out an XBEL file in each generated directory that contains the XBEL fragment for the corresponding folder. You can apply XSLT transformations to these as well. I'm currently using this to generate OPML [3] as well as XHTML representations of the folder index. So the concept of treating each Folder as a partitioning device exists here too.

At the end of the day, you end up with a system that takes an XBEL document as input and produces a directory structure containing normalized / converted feeds as well as aggregated index feeds. These are all simple files and directories so exposing via your favorite web server is straightforward. I've written XSLT for RSS 0.91, RSS 1.0/RDF, and XHTML on the feed side and OPML and XHTML on the index side. It is pretty trivial to plug in new representations for both feeds and indexes given an XSLT that takes an Atom 0.3 document on the source side.

This is all very experimental right now and while I'm using the system for day-to-day reading, I'm also breaking stuff and performing major restructing of code and concepts very often. The system is definitely not without its problems. I plan on blogging the success and failure of various approaches in moderate detail over the next couple of months. I haven't even put a distribution together yet but please feel free to browse the sources or grab a tarball if you're interested in really early, often broken applications. I will have to find a home for the project eventually (I'm trying to avoid sourceforge if possible) as all of this is hosted off of a P300 sitting in my living room with only a humble Road Runner pipe. I imagine I will get more serious about this when I think of a name or the code starts stabilizing, whichever comes first. In the meantime, please leave comments or shoot me an email if your interested.

[1] XSLT : http://www.w3.org/TR/xslt
[2] XBEL : http://pyxml.sourceforge.net/topics/xbel/
[3] OPML : http://opml.scripting.com/

Enter movabletype. I loaded this up at work so I could keep a web based research journal. I was awed by how simple it was to setup and the amount of functionality provided right out of the box. This thing does its job and gets out of your way. So anyway, long story short, pyblosxom's out for a little while (until the community grows and development starts up again) and movabletype is in.

# Do the right thing with boolean values for all known Python versions.
try:
  True, False
except NameError:
  (True, False) = (1, 0)


I should note here that apt-get can be used on RPM based systems as well but I am apt-get ignorant and will be refering to Yum exclusively for the rest of this entry. It is extremely possible that apt-get or even up2date could provide the the functionality needed to restore a system from RPM.

More pertinent to the topic at hand is the fact that Yum takes care of pulling RPMs from remote repositories, working out dependencies, and performing installations given a list of package names. What this means to the backup artist is that in order to be able to restore a wrecked system to it's previous set of packages you need only backup your yum config file (usually /etc/yum.conf), which contains your repository configuration, along with the names of all packages installed on your system. We can get the list of packages easily enough:

$ rpm -q --all > list-of-packages

Once we have our yum.conf and package-list backed up, a fresh machine with just the minimum requirements to run Yum, should be able restore to it's previous state with something like the following:

$ cat list-of-packages | xargs yum install

You have to swish the concept around in your head a little bit but you can think of the many RPM repositories that are getting thrown up nowadays as shared backups for common stuff. This greatly reduces the amount of files that need to be backed up by each person because they can always be obtained from a publicly available source.

So we can backup every single RPM managed file [that hasn't been modified] with a very small footprint. Now we get into the more tricky part, which is backing up all the stuff RPM either doesn't manage or manages but determines has been modified. We will use rdiff-backup to increment the files for all the benefits stated previously, but there are a few things missing from the current rdiff-backup/yum toolset.

• There is no straightforward method of obtaining a list of files that are not managed by RPM. You can rpm -ql --all to get a list of files that are managed by RPM and feed them into rdiff-backup as an exclude list but there are issues with this. Most notably is the fact that the file list will contain directories managed by RPM, which may contain files/directories not managed by RPM and rdiff-backup will exclude all files in a directory if supplied in an exclusion list.
• rpm -ql -all outputs full filenames relative to the root of file system. This means that rdiff-backup would always have to be run on / as file paths in the exclude list are interpreted relative to the directory being backed up.
• While not impossible, it is fairly annoying to get a list of files that are managed by RPM but have been modified from their original versions. The following is RPM's verify output, which can be parsed to find modified files:

  $ rpm --verify --all --nomtime --nodigest \
       --nosignature --nodeps --nordev \
      --noscripts --nouser --nogroup --nomode
S.5..... c /etc/ant.conf
S.5..... c /etc/krb.conf
S.5.....   /usr/lib/rpm/rpmrc
S.5..... c /etc/X11/gdm/gdm.conf
...

  This has some serious drawbacks including the fact that it will take a very long time to run on even the fastest machine because it's performing MD5 checksums on thousands files, which also pretty much makes the machine unusable. You can speed things up a bit by passing --nomd5 into rpm --verify, which will cause checking to occur on sizes only but there's a pretty good chance you will miss something.
• Lastly, it would be nice if files known to have no value in a backup could be excluded by default. e.g. /var/run, /var/lock, /tmp, /home/*/.mozilla/*/*/Cache, etc.

What we need is a library that, given a directory, will tell us what files and sub-directories are unmanaged or managed-but-modified. This tool should be able to utilize some kind of database or cache of file information (possibly slocate's database or rpm's).

Once we have a mechanism for determining what files are unmanaged, a nice framework for defining backup sets should be provided. I'm thinking something along the lines of having an /etc/backup.d directory that would have config files for each backup set. The config files would specify the root directory to backup, where the backups should be stored, and a list of additional exclusions and/or forced inclusions relative to the backup directory.

# sample backup config file
[info]
name=home directories
source=/home
dest=root@backup-host:/backups/$hostname/home
backuptype=rdiff-backup
frequency=1d                # how often should we backup?
retain=5d                   # how long should increments be
                            # kept around?

[files]
- .phoenix/**/Cache         # exclude mozilla firebird cache
- .Xauthority               # we don't need that either..
+ .Xresources               # forcibly include .Xresources

A couple of concepts to point out here. Source and dest are pretty self explainatory. backuptype would allow other backup tools to be used in place of rdiff-backup. Maybe we just want to straight rsync the files, or maybe somebody feels that incremental tarballs are still relevant <g>. Lastly, the files list contains a list of files to include or exclude. I really like rdiff-backups file selection syntax ; it is simple and powerful.

There should also be a sane default backup set containing all unmanaged/modified files on the system minus cruft like /var/{run,lock}, cache directories, and anything else that doesn't have backup value. The only aspect that should require configuration is the destination of the backup. Reasonable defaults for all other aspects should be provided.

A Minimal Backup System would provide a simple-as-in-easy-to-configure yet powerful tool that could act as an almost turnkey backup solution for most small scale GNU/Linux installations that use RPM for package management. A positive side-effect of having RPM aware backups is that it further promotes the use of RPM to package common/unchanging files as the more that is managed by RPM the less space is required for backups.

• uptime
• nukeimage
• useragentswitcher
• popupcount
• autohide
• rssreader

At this point I'm fairly confident that any extension can be packaged in RPM. I'm planning on soliciting feedback from the fedora and freshrpms lists soon and should be publishing something in the interim.

If you want to take a look at building these yourself, I have viewcvs setup here with the tarball option on.