About Tom Evslin

Video Profile of Tom Evslin

Follow Tom Evslin on Twitter


subscribe:

Add to Technorati Favorites!
Powered by TypePad
Member since 01/2005

technorati


« Venture Capital – An Entrepreneur’s View (continued) | Main | The Search Bee »

The Flattening of Almost Everything #2: Information Retrieval

The WorldWideWeb is where Moore’s Law met Metcalfe’s Law.  Information management – the way we find out what we want to know – went from hierarchical to flat in just a few years as a result.  We now assume – usually correctly – that we can find any particular piece of data from a railroad schedule in Estonia to a quote by an Argentine novelist on the Web within minutes of wanting it.  We also rely on the web for cross–references (links)  to interesting information related to whatever we originally searched for specifically.

Back when I was at Microsoft, Lotus Notes, written by the brilliant Ray Ozzie, was the competitor which worried Bill Gates (and, therefore, the rest of us) the most.  Companies were building information management application in Notes.  True, Notes ran under Windows, but the danger we saw was that Notes and not Windows would be the platform that developers wrote to.  Many of the pundits were saying (hoping) this would happen.

There were several competing efforts under way in Redmond to build the Notes-killer.  One of them was mine: Microsoft Exchange Server.  Exchange was behind schedule for release when I took it over and slipped even further as we tired to shoehorn in features that would one-up or at least match the information-handling capabilities of Notes.  Trouble was that Exchange was also the long-overdue replacement for DOS-based Microsoft Mail.

Another effort was Cairo (a future release of NT) championed by Jim Allchin of Banyan Vines fame.  Here the information was managed at the operating system level rather than in the email server.  The database guys had their own effort underway.  “Ren and Stimpy” was the code name for Brian MacDonald’s brilliant concept in a personal information manager (PIM) which eventually became the Outlook client.

We all argued long and hard and as loudly as only Microsoft people can do about which of these was the correct solution, which should own the APIs used by Office for information management, and which ideas were brain dead.  Bill kept the competition alive by not deciding between us.  I think he wanted to see what emerged.

But every one of these solutions – including the bogyman, Notes – was hierarchical.  There were folders within folders within folders.  Sure, there were key word searches.  And categories could be assigned.  Different views could be produced.  But we all assumed that most people would approach information through the categories they assigned the information to.

To put it mildly, we were all wrong!

The WorldWideWeb, the sea of information that we now can’t imagine living without, is flat!  “Flat” isn’t even really a good metaphor – the WorldWideWeb is actually dimensionless.  You can navigate directly from any page to any other page.  Any page can point to any other page.  And, although websites are nominally hierarchical, search engines and links point you directly to the page on the site that you are looking for.

The power of this horizontal approach to information doomed Notes to an increasingly irrelevant niche, sent Exchange back to its proper role as an email system, and Outlook to its role as an email/scheduling/tasks client.

People don’t think hierarchically – at least most people don’t.  We think in terms of associations.  Our dreams give this away as they hyperlink through experiences of the day and memories of the distant past.  A conversation meanders horizontally from one topic to the next. “That reminds me of…” is the way we get from one place to another in our own brains.  Some day we may understand this mechanically as an obvious consequence of the way neurons connect.

Hierarchies like Lotus Notes or the Dewey Decimal System were necessary when computing power was non-existent or very expensive. As computing power has become relentlessly cheaper thanks to Moore’s law, hierarchies of information have become unnecessary.  Cheap MIPs made graphical browsers,  higher-bandwidth modems, smart lights in fiber, and search engines possible.  All that we needed then was the WorldWideWeb so that almost all information became available to the galloping bots and hierarchies of information became obsolete.  So long as Google or its competitors can index almost everything I might ever want to find, why should any arbitrary order be imposed on information?  In fact, Metcalfe’s law, which states that the value of a network is proportional to the square of the number of endpoints, may be an understatement when applied to a hyperlinked network where there can be value in multiple references between any pair of documents.

Once we didn’t need hierarchies to organize our approach to information, they became an impediment.  It is very hard for one person to figure out which node in which folder tree another person would have put a particular piece of information.  A document may be relevant to one researcher for entirely different reasons than it is relevant to another researcher.  The creator of a document doesn’t know all the ways the information in the document may be used. 

The relationship between documents is actually dynamic depending on the needs of the reader.  Not incidentally, open tagging and hyperlinking are both ways to impose particular relationships on documents to meet the need of some subset of readers. These relationships, themselves, can and do evolve constantly on the web.

The flattening of the information space is part of an accelerating and self-reinforcing trend of change.  The flattening was enabled by the two great inventions of the Internet and personal computers.  But, with information now much more readily available than it has ever been before, innovation becomes easier and change continues to accelerate.

See Search Bees for educational implications of this change.

Previously, I blogged on the flattening of organizational hierarchies.  Related to that is the demise of vertical integration and implications for telco mergers.

The flattening of bureaucracies is a particularly satisfying special case of hierarchies being flattened.  I blogged about some hopeful signs in India.

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d83451cce569e200d834230d4353ef

Listed below are links to weblogs that reference The Flattening of Almost Everything #2: Information Retrieval:

» The world is flat from you can sleep when you're dead
Tom Evslin (creator of Microsoft Exchange Server) explains why tagging is more powerful than foldering The relationship between documents is actually dynamic depending on the needs of the reader.  Not incidentally, open tagging and hyperlinking ar... [Read More]

» IT, Beauty and Grace from A Nation of the People and IP Addresses
One of the reasons that I'm an optimist when it comes to the use of technology is that it supports our natural thought processes. It’s associative, intuitive and connects many of the random associations that come, say, in the middle [Read More]

» Words, Words, Words from Blogalicious -- Technology and Information Society -- Steve Damron's Blog
The blogosphere is full of language technology articles today. Wired writes on how the web is not the end of language. [Read More]

Comments

Dag Kvello

So, I guess You never (still don't) did grasp the concept of Lotus Notes.

It never was (and isn't) hierarchical. It is flat (dimensionless) and unstructured.

Presentation has allways been seperated from content.

Full-text searching has allways been at the core (since nothing is stored in a structured manner).

Hierarchy is something You only create in the presentation-part and usually only to present the unstructured data in structured manners.

Actually Notes works as the Internet, only more advanced.

TomNorian

OK...soap boxing a bit more.

Fast and slow are the same subject: "speed"

But Fast also means obstaining from food or tieng (got no clue how to spell tie with an ing at the ent( something tight.

Speed, Spead, Spede? It is a shame that we need to put our concept to words but I agree inecessary hurdle I am faced with.

What I have a harder time with is understanding relationships between Fast Slow Scamper Stagger.

Are "scamper and fast" more related or "scamper" and "stagger"

One concept is speed and the other is of spirit and remaining energy?

Steve Castellano

Very interesting post. I was thinking much of the same when I wrote about information commodization (see link below). As the cost of acquiring information declines, there will be increasing value placed on providing analysis of that information.

http://roer.blogs.com/my_weblog/2004/11/commoditzation_.html

Johannes Ernst

And then we got XML ... and a "new" generation of developers grows up believing that hierarchical data representation is the only right way.

You can make the same argument on a database level: hierarchical databases gave way to relational databases, and for good reasons.

Tom Evslin

Mark:

I think you're onto something. Transient categories certainly have a use; transient hierarchies may. It would be great if you could provide an example of a need met by a hierarchy that can't be met by rapid search and catgorization.

In any case, the key is that we should be able to have our agents build these taxonomies as transient structures giving a view of information rather than permanently trapping the information in a particular taxonomy.

Mark Crowne

Tom,

Thanks - very interesting. As someone interested in content management this stimulated a thought about separation of content and presentation.

Sure, storage is flat for the reasons that you state. But perhaps I want my view of information to be heirarchical in a way that makes sense to me, allowing me to browse it as well as search it.

Maybe what we need is an agent that searches the information, picks out stuff that I''m interested in, and organises it hierarchically in my preferred taxonomy?

I guess we'd need some kind of fuzzy taxonomy as well, to make this work. Perhaps this could be extracted from the heirarchy in which I choose to store my information. Maybe the same agent could tell me when I tried to store some information that I was putting in the wrong place according to my own taxonomy.

Regards

Mark

PG

Hello Tom,

Now, I know some librarians, expert taxonomists, etc...who are going to have a very hard time with this...me, on the other hand, I love it because it's so simple.

Post a comment

If you have a TypeKey or TypePad account, please Sign In.

Now on Kindle!

hackoff.com: An historic murder mystery set in the Internet bubble and rubble

CEO Tom Evslin's insider account of the Internet bubble and its aftermath. "This novel is a surveillance video of the seeds of the current economic collapse."

The Interpreter's Tale

Hacker Dom Montain is in Barcelona in Evslin's Kindle-edition long short story. Why? and why are the pickpockets stealing mobile phones?

Need A Kindle?

Kindle: Amazon's Wireless Reading Device

Not quite as good as a real book IMHO but a lot lighter than a trip worth of books. Also better than a cell phone for mobile web access - and that's free!

Recent Reads - Click title to order from Amazon


Google

  • adlinks
  • adsense