My (Yahoo) Pipe Dream

With all the entertainment options available nowadays, I constantly feel like I’m missing something. Unfortunately the online solutions for keeping track of what’s going on in Prague are few and far between, and they require that you visit a slew of different websites one by one. RSS use is increasing very slowly, I suspect because so many sites depend on advertising and thus on people actually visiting their domain.

My dream is a website a la Techmeme where I can see an overview of all the things going on over the next few days that might interest me: new film releases, golf and tennis on TV (and big matches here in Prague… a rare occurrence), concerts, new video downloads, new podcasts and the occasional artsy happening (gallery opening, museum exhibition, theater, modern interpretive dance, etc.) in case I happen to be feeling particularly sophisticated. (Actually forget about the modern interpretive dance. I never feel that sophisticated.) I also want to see at a glance the weekend’s weather in the various local golfing hotspots. Obviously there has to be a monster filter operating to keep me from getting overwhelmed. Ideally it would learn from my preferences and prioritize events based on how likely I am to be interested.

Since nothing like this is going to exist any time soon, at least not for Prague, I’ve decided to build it myself in my “spare time” while trying out some of the latest greatest web tools. This might take years considering the time I have to devote to it, but I did manage to take the first baby steps last weekend while pretending to watch a fairly boring golf tournament on Sunday evening. This involved nothing more than constructing a customized RSS feed of new movie releases combined with IMDb ratings.

IMDb doesn’t have any kind of web API (would someone please make a wikified open source movie database with user ratings?), so I decided to try OpenKapow as a way of scraping the data into something machine processable. I downloaded their massive Java-based design tool, but never got to use it since I found an existing robot that does exactly what I wanted.

Then I scooted over to Yahoo Pipes, something I’ve been wanting to try for ages. My impressions:

• The user interface is amazingly slick. This is AJAX at its smooth-scrolling dragging-and-dropping best.
• The programming paradigm is a bit disorienting for a old-school procedural hacker like myself, but it’s clean and consistent, and when it starts to make sense it’s pretty darn nifty.
• They could use far more tutorials, examples and documentation. One example for each little widget, with no overview of the programming model that I could find, is painfully insufficient.
• The forums are great. There appears to be a group of passionate and knowledgeable users who are keen to help out struggling newbies.
• It works! The service was apparently suffering some hiccups when I started playing around with it, which was a bit frustrating, but people on the forums assured me that this was a very rare occurrence. When I tried it again a couple of days later I encountered no problems whatsoever.
• I can’t see this scaling as a centralized service. All of the filtering and mashing up should happen on the client. I’d love to see this as part of AllPeers at some point in the (probably rather distant) future. I suspect the glitches I encountered were to due to overload, and just now I received a weird error message when I tried to published the final version of my pipe.

My pipe is now published online for the world to see. Next steps are to try OpenKapow on some local information sources and then take the various personalized homepage services for a spin by streaming my RSS feeds into them. I also want to put Particls (whose RSS filtering technology is in the news today) through its paces. Stay tuned for (infrequent) updates.

Bookmark to:

2 Comments »

1. I checked out your pipes. I can see a few optimizations. In your title lookup pipe, you defined a default title. It seems to me that your purposes would be better served by putting the title in the debug field. In the default feild, calling the pipe without input will yeild the value for the default. In your regex in the main module you seach for (.*) to capture the entire contents so that you can append a string to the title. Another way to do this is to match $ - the metacharacter for the end of a string - no need to capture what is there. You can match the beginning of s string with ^.

   Comment by David Robarts — 5/29/2007 @ 7:40 pm

2. Thanks David, I’ll make the changes you suggest.

   Comment by Matt — 5/30/2007 @ 4:20 pm

Trackback URL RSS feed for comments on this post.

Leave a comment

Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>