TITLE: Design Ideas Lying in Wait AUTHOR: Eugene Wallingford DATE: August 07, 2008 2:57 PM DESC: ----- BODY: Ralph Johnson pointed me to a design idea for very large databases called a shard. This is a neat little essay for several reasons. First, its author, Todd Hoff, explains an architecture for massive, distributed databases that has grown up in support of several well-known, high-performance web sites, including Flickr, Google, and LiveJournal. Second, Hoff also wrote articles that describe the architectures of Flickr, Google, and LiveJournal. Third, all four pages point to external articles that are the source of the information summarized. Collectively, these pages make a wonderful text on building scalable data-based web systems. I've posted this entry in my Patterns category because this recurring architecture has all the hallmarks of a design pattern. It even has great name and satisfies Rule Of Three, something I've mentioned before -- and what a fine three it is. Each implementation uses the idea of a shard slightly differently, in fitting with the particular forces at play in the three companies' systems. Buried near the bullet list on the Google page was an item worth repeating:
Don't ignore the Academy. Academia has a lot of good ideas that don't get translated into production environments. Most of what Google has done has prior art, just not prior large scale deployment.
This advice is a bit different from some advice I once shared for entrepreneurs, looking for Unix commands that haven't been implemented on the web yet, but the spirit is similar. Sometimes I hear envious people remark that Google hasn't done anything special; they just used a bunch of ideas others created to build a big system. Now, I don't think that is strictly true, but I do think that many of the ideas they used existed before in the database and networking worlds. And to the extent that is true, good for them! They paid attention in school, read beyond their assignments, and found some cool ideas that they could try in practice. Isn't that the right thing to do? In any case, I recommend this article and encourage others to write more like it. -----