The Vertical Search Engine Debate

Whether or not “vertical search engine” is an oxymoron, as I blogged last week,  depends on how you define search engine.  There have been many intelligent rebuttals to my blog including those by Jupiter’s Niki Scevak, who was lead analyst on the Jupiter report with which I disagreed, and by Fred Wilson in his blog.  Niki was kind enough to send me a copy of the full report gratis so that I would be better informed.

Fred says he would use Kayak to make reservations, not Google.  I agree that Google is not the best place to make a plane reservation.  But I wouldn’t classify Kayak as a vertical search engine.  It is a provider of  specific set of services with a structured database.   Kayak doesn’t crawl the web to find seats on planes; it uses feeds directly or indirectly from airlines.  Similarly, a job-finding site’s technical value lies in the fact that it aggregates structured data and provides a structured interface for finding the right candidate or the right job – it obviously has a community value as well.

I did say: “Specialized sites will continue to have specialized search interfaces for their own content – particularly if the content itself is inherently structured”; but, judging from the response, I didn’t make clear enough what I meant by “specialized sites”.  Lack of clarity doesn’t go unchallenged long in the blogosphere.  To be clear, I do think that there will be a proliferation of specialized databases and structured interfaces for searching them.  From an advertiser’s point of view, I even agree with Jupiter that these specialized sites need to be considered along with horizontal search engines in putting together a product campaign.

Brad Burnham point out the usefulness of communities as filters to help determine the relevance of search results seen from a particular point of view.  Again, I agree and should have been more clear about this.  I see these communities as value added on top of horizontal search, however, not a competitor to it.  The communities are certainly a distinct placement opportunity for advertisers.

As if to illustrate some of the weaknesses of keyword-based advertising, I was surprised to see AdSense placing ads for Franklin Roosevelt memorabilia on my blog.  It took a few minutes to figure out that this was because my newest post was about Regulation FD (as in FDR).  Didn’t get many clicks on these.

The horizontal search engines are semantic based and, I think, the best way to locate unstructured textual content on the web.  When I was in college a million years ago I worked on an NSF-funded project to compare search algorithms  - not because it had anything to do with my history and literature major but because programming paid better than my other skills of room cleaning, bartending, and snow shoveling.  To our surprise, we found that almost no value was added by analyzing the syntax of a query or a document.  Allowing simple boolean relationships between search terms – AND, OR, NOT – did help. 

Since our search engines had to pass reels of mag tape on an IBM 7094 for each query, they were hardly ready for prime time.  Nevertheless, horizontal search engine development seems to bear out our results.  The words in a query are matched to the words in a universe of documents – boolean filtering allowed if desired – and ranked results are returned.  We even found that the option of “more like these?” was useful in refining searches.

But, thanks to the comments from the blogosphere, I now realize that structured search as in a travel, job, or dating service, is a substitute for syntax when the data itself is structured.  Dropdowns, radio buttons, check boxes etc. are an easier way for people and machines to communicate than trying to phrase and parse the sentence “I would like two business class seats on a flight from New York to Los Angeles on April 1, 2005 leaving as early in the morning as possible.  I would like direct flights only and please optimize for schedule rather than price.  I am willing to use any New York airport but would prefer to go into LAX.”

Nevertheless, I strongly disagree with search engine marketing guru Danny Sullivan who is quoted on ClickZNews as saying "I can't say it enough. Vertical search is going to take over."  I wasn’t there to hear his talk so he may have had some convincing argument; but there isn’t a chance in the world that vertical search – even if defined to include what I would rather call structured search – is going to displace the horizontal search engines. Structured search with its artificial syntax is a great supplement to horizontal semantic search.  Structured databases are a needed adjunct to the unstructured documents which comprise the bulk of the web.   But horizontal search of the whole accessible web will continue to be the starting point for finding almost everything including the structured search engines themselves.

I also continue to disagree with Jupiter’s assertion that the web will become more like television with its myriad vertical channels.  To be fair, this point is stronger in the promo for the Jupiter paper than it is in the paper itself.  Om Malik provides great evidence of TV becoming more like the web in his post about startup Brightcove and its platform for bypassing channels as an intermediary between producers and consumers of video content.