TITLE: Design Ideas Lying in Wait
AUTHOR: Eugene Wallingford
DATE: August 07, 2008  2:57 PM
DESC: 
-----
BODY:
<A HREF="http://st-www.cs.uiuc.edu/users/johnson/">
   Ralph Johnson</A>
pointed me to a design idea for very large databases
called a
<A HREF="http://highscalability.com/unorthodox-approach-database-design-coming-shard">
   shard</A>.
This is a neat little essay for several reasons.  First,
its author, Todd Hoff, explains an architecture for massive,
distributed databases that has grown up in support of
several well-known, high-performance web sites, including
Flickr, Google, and LiveJournal.  Second, Hoff also wrote
articles that
describe the architectures of
<A HREF="http://highscalability.com/flickr-architecture">
   Flickr</A>,
<A HREF="http://highscalability.com/google-architecture">
   Google</A>,
and
<A HREF="http://highscalability.com/livejournal-architecture">
   LiveJournal</A>.
Third, all four pages point to external articles that are
the source of the information summarized.  Collectively,
these pages make a wonderful text on building scalable
data-based web systems.

I've posted this entry in my
<A HREF="http://www.cs.uni.edu/~wallingf/blog/archives/cat_3.html">
   Patterns</A>
category because this recurring architecture has all the
hallmarks of a design pattern.  It even has great name and
satisfies
<A HREF="http://c2.com/cgi/wiki?RuleOfThree">
   Rule Of Three</A>,
something I've
<A HREF="http://www.cs.uni.edu/~wallingf/blog/archives/monthly/2004-08.html#e2004-08-29T16_29_16.htm">
   mentioned before</A>
-- and what a fine three it is.  Each implementation uses
the idea of a shard slightly differently, in fitting with
the particular forces at play in the three companies'
systems.

Buried near the bullet list on the Google page was an item
worth repeating:

<BLOCKQUOTE><EM>
Don't ignore the Academy.  Academia has a lot of good ideas
that don't get translated into production environments.
Most of what Google has done has prior art, just not prior
large scale deployment.
</EM></BLOCKQUOTE>

This advice is a bit different from some advice I once shared
for entrepreneurs,
<A HREF="http://www.cs.uni.edu/~wallingf/blog/archives/monthly/2007-03.html#e2007-03-30T18_51_07.htm">
   looking for Unix commands</A>
that haven't been implemented on the web yet, but the spirit
is similar.  Sometimes I hear envious people remark that
Google hasn't done anything special; they just used a bunch
of ideas others created to build a big system.  Now, I don't
think that is strictly true, but I do think that many of the
ideas they used existed before in the database and networking
worlds.  And to the extent that is true, good for them!  They
paid attention in school, read beyond their assignments, and
found some cool ideas that they could try in practice.  Isn't
that the right thing to do?

In any case, I recommend this article and encourage others
to write more like it.
-----