A Perfect Swarm

Are proprietary standards doomed to extinction in today’s open-source world? Brad Neuberg, an open source and P2P developer, apparently thinks so, and he explains why in a provocative blog entry arguing that SwarmStream, a commercial P2P product, is going to be trounced by BitTorrent, its wildly popular open source competitor. What makes this post particularly intriguing is that SwarmStream CEO (and inventor of the swarming concept), Justin Chapweske, takes the bait and defends his company’s policies, leading to a well-argued dialogue that exposes both sides of the commercial vs. open source debate.

Some background is probably in order. Swarming is a clever way of transferring files without creating bottlenecks as load increases. Instead of having one or more servers that are responsible for sending files to any and all downloaders, every computer that is downloading immediately becomes a server in its own right and starts sending the bits that it already has to other users. In a classic client/server architecture, most computers in the network are downloading like crazy but not uploading anything. Since most internet connections are more or less symmetrical (i.e. you have the same bandwidth available for download and upload), this leads to big waste of aggregate bandwidth. Swarming, so named because it creates a “swarm” of computers all sending data back and forth, harnesses this wasted upload potential. The result is that popular files actually download faster than unpopular ones since there are more people making them available.

BitTorrent exploaded on the scene in 2003 with the first really usable implementation of swarming. It quickly became the file sharing software of choice for larger media files like TV shows and movies, as well as being adopted by the geek clique for distributing large software packages like Linux. It’s succeeded because it works (newer versions of BitTorrent clients have had to include a feature to limit download speed since file transfers were so fast that they frequently maxed out a computer’s internet connection). But it’s not perfect. One significant flaw is that files are divided into chunks that are distributed more or less at random to a downloading machine, so you don’t necessarily get the beginning of a file first. This makes it impossible to do streaming via BitTorrent, so you have to wait for the entire file to download before you can view it.

As the name implies, SwarmStream’s inventors have created their own swarming protocol with all the advantages of BitTorrent as well as the ability to stream files for real-time viewing. The product also includes enterprise-level features like secure encryption. There’s just one catch: unlike BitTorrent, which is free and open source, SwarmStream is a commercial product with a $25,000 price tag.

This leads us to Neuberg’s rant. In essence he claims that due to its proprietary nature SwarmStream will never become a standard. Meanwhile, BitTorrent can draw on armies of open source developers to improve it until it catches up technically, at which point customers, given the choice between paying nothing or paying $25,000, will desert SwarmStream in droves. Chapweske, on the other hand, argues that his company can make plenty of money off corporate clients without becoming a standard. Free software advocates are only on his case, he says, because they’re lusting for his technology but can’t afford to license it.

If industry precedent is any guide, Neuberg has the stronger case. The example of RSS (a ubiquitious news feed protocol) versus ICE (a now forgotten commercial competitor) is apt, and it’s not hard to find other examples of open technologies that have beaten out their closed brethren. Start with the web itself, which easily saw off attempts by the likes of Microsoft and AOL to create “walled gardens” on the internet. Linux has become an immediate threat to proprietary Unix systems like Sun Microsystem’s Solaris. Meanwhile, the threat of open technologies has forced Microsoft, that bastion of proprietary control, to relinguish ownership of VBScript (now ECMAScript) and ActiveX to independent standards bodies

The software industry has been transformed by the advent of the internet because network effects play a much larger role now that we have ubiquitious connectivity. Companies continue to dream of “pulling a Microsoft” by establishing a de facto standard while maintaining control over the associated intellectual property. But in an age where even Microsoft is hedging its bets in this regard, it is more apt to regard the Windows monology as an anachronism. The Microsofts of today are companies like Amazon and Google that have succeeded mainly through the consistent excellence of their products, not any implicit “lock in” that forces existing users to stick with them. Other players, such as eBay, Adobe and MySQL have managed to establish de facto standards by offering their core products for free while charging for certain types of usage.

So Neuberg is right: SwarmStream might make a few bucks in the interim, but it is never going to establish a standard with a base price tag of $25,000, and so is doomed to defeat at the hands of a more open competitor, be it BitTorrent or someone else. However: this hypothesis assumes that SwarmStream fails to take action to address this issue, and Chapweske states explicitly that “It has always been my intent to allow everyone access to this technology, so keep your eyes peeled to my blog…” A few years ago, Neuberg’s griping might have been rejected out of hand as the ravings of a naive hippie. The fact that even purveyors of commercial software now recognize the need for free and open software in order to win in the marketplace is a sign that proprietary standards are truly becoming a thing of the past.

Bookmark to:

1 Comment »

1. Hi Matt; nice blog post. I’ve updated my original essay to point to your response. Scroll to the bottom at http://codinginparadise.org/weblog/2004/12/why-bittorrent-is-important-and.html to see the link to this post.

   Thanks,
   Brad

   Comment by Brad Neuberg — 12/20/2004 @ 9:59 pm

Trackback URL RSS feed for comments on this post. TrackBack URI

Leave a comment

Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>