The Vertical Search Engine Debate

Whether or not “vertical search engine” is an oxymoron, as I blogged last week,  depends on how you define search engine.  There have been many intelligent rebuttals to my blog including those by Jupiter’s Niki Scevak, who was lead analyst on the Jupiter report with which I disagreed, and by Fred Wilson in his blog.  Niki was kind enough to send me a copy of the full report gratis so that I would be better informed.

Fred says he would use Kayak to make reservations, not Google.  I agree that Google is not the best place to make a plane reservation.  But I wouldn’t classify Kayak as a vertical search engine.  It is a provider of  specific set of services with a structured database.   Kayak doesn’t crawl the web to find seats on planes; it uses feeds directly or indirectly from airlines.  Similarly, a job-finding site’s technical value lies in the fact that it aggregates structured data and provides a structured interface for finding the right candidate or the right job – it obviously has a community value as well.

I did say: “Specialized sites will continue to have specialized search interfaces for their own content – particularly if the content itself is inherently structured”; but, judging from the response, I didn’t make clear enough what I meant by “specialized sites”.  Lack of clarity doesn’t go unchallenged long in the blogosphere.  To be clear, I do think that there will be a proliferation of specialized databases and structured interfaces for searching them.  From an advertiser’s point of view, I even agree with Jupiter that these specialized sites need to be considered along with horizontal search engines in putting together a product campaign.

Brad Burnham point out the usefulness of communities as filters to help determine the relevance of search results seen from a particular point of view.  Again, I agree and should have been more clear about this.  I see these communities as value added on top of horizontal search, however, not a competitor to it.  The communities are certainly a distinct placement opportunity for advertisers.

As if to illustrate some of the weaknesses of keyword-based advertising, I was surprised to see AdSense placing ads for Franklin Roosevelt memorabilia on my blog.  It took a few minutes to figure out that this was because my newest post was about Regulation FD (as in FDR).  Didn’t get many clicks on these.

The horizontal search engines are semantic based and, I think, the best way to locate unstructured textual content on the web.  When I was in college a million years ago I worked on an NSF-funded project to compare search algorithms  - not because it had anything to do with my history and literature major but because programming paid better than my other skills of room cleaning, bartending, and snow shoveling.  To our surprise, we found that almost no value was added by analyzing the syntax of a query or a document.  Allowing simple boolean relationships between search terms – AND, OR, NOT – did help. 

Since our search engines had to pass reels of mag tape on an IBM 7094 for each query, they were hardly ready for prime time.  Nevertheless, horizontal search engine development seems to bear out our results.  The words in a query are matched to the words in a universe of documents – boolean filtering allowed if desired – and ranked results are returned.  We even found that the option of “more like these?” was useful in refining searches.

But, thanks to the comments from the blogosphere, I now realize that structured search as in a travel, job, or dating service, is a substitute for syntax when the data itself is structured.  Dropdowns, radio buttons, check boxes etc. are an easier way for people and machines to communicate than trying to phrase and parse the sentence “I would like two business class seats on a flight from New York to Los Angeles on April 1, 2005 leaving as early in the morning as possible.  I would like direct flights only and please optimize for schedule rather than price.  I am willing to use any New York airport but would prefer to go into LAX.”

Nevertheless, I strongly disagree with search engine marketing guru Danny Sullivan who is quoted on ClickZNews as saying "I can't say it enough. Vertical search is going to take over."  I wasn’t there to hear his talk so he may have had some convincing argument; but there isn’t a chance in the world that vertical search – even if defined to include what I would rather call structured search – is going to displace the horizontal search engines. Structured search with its artificial syntax is a great supplement to horizontal semantic search.  Structured databases are a needed adjunct to the unstructured documents which comprise the bulk of the web.   But horizontal search of the whole accessible web will continue to be the starting point for finding almost everything including the structured search engines themselves.

I also continue to disagree with Jupiter’s assertion that the web will become more like television with its myriad vertical channels.  To be fair, this point is stronger in the promo for the Jupiter paper than it is in the paper itself.  Om Malik provides great evidence of TV becoming more like the web in his post about startup Brightcove and its platform for bypassing channels as an intermediary between producers and consumers of video content.

Comments

[Danny Sullivan]

No, "horizontal" search engines won't get wiped out by vertical ones. They'll simply become more vertical in how they present horizontal results. That's what I was saying in my talk. To understand it better, see this: http://searchenginewatch.com/searchday/article.php/3115131

[Adam Gedde]

Tom:

You raise a very interesting point. A point that, in my opinion, has rarely been discussed when addressing the issue of "vertical search".

I think we would all agree that horizontal search is the best and most widely used way to find specific information in the generalized space that is the Internet. Certainly, the improvements in semantic algorithms that the "Big 3" search engines have implemented have made finding the preverbial needle in the haystack much easier for the majority of the web population.

So why all the fuss about Vertical Search? I think there is a strong case to be made that Yahoo, way back when, was a structured, searchable directory which relied on humans to place content in its appropriate category. These days, Yahoo still has their directory, but it's been augmented/supplanted by its search technology.

Therefore, using Yahoo as an example, it could be said that the concept of vertical search is nothing new. Yet here we are - companies and marketers touting vertical search as the next big thing.

I enjoyed your summation of the differences between semantic and structured search, and I think that is the key reason why the vertical movement has become a hot topic. Semantic search, while it provides some of the most relevant information does not account for the ways in which most people would like to search for that information.

Davi put it succinctly: semantic = searching for knowledge and vertical = searching for stuff.

In order to have the granular control needed to find the "stuff" people look for, there needs to be a fair amount of structure that the semantics cannot provide. Verticals can provide that because their datasets have the structure necessary to drill down into specifics.

Most VSE's are, and will remain, structured. In this sense they are not going to be a true search engine. They will not be able to turn their algorithms loose on the web and expect to come back with information for their market space. They will, however, continue to rely upon structured data fed in from multiple parties and organized in a searchable fashion.

Yet, there are those who are making inroads into the true vertical spaces. As you said - Froogle is Google's attempt at doing so.

On a much smaller scale than Google, Fetchster.com, a Minnesota Job Search Engine of which I am the founder/CEO, has taken the approach that spidering/crawling can work, and that there can be a synergy between semantic and structured search.

I will admit that Fetchster.com is not a true semantic search engine. However, none of our content is fed in from outside providers. Our search software spiders the content on the employment portals of Minnesota companies, structures it, and brings it back into a searchable database. As Om Malik put it - "Fetchster, even more vertical (job) search" http://gigaom.com/2005/05/16/fetchster-even-more-vertical-job-search/.

On the frontend it's still a structured search for our customers. It has to be. But, the way in which we gather our data is an important first step towards the holy grail of semantic search within the "vertical" market.

Adam Gedde
http://www.fetchster.com/

[Davi Gibbons]

Q? - when is a search not a search?
A? - when it's a vertical search! (or possibly, a search for 'stuff')

Tom, you are absolutely correct in this; your critics, imo, are playing with symmantics. Which side of this discussion you stand on depends exclusively on your definition of "search"! If you define search as an ad-hoc quest for knowledge, it's absolutely horizontal - or trending that way - no doubt.

In "searching" for things other than general knowledge, like airline reservations (which are a poor comparison), you obvoiusly wouldn't go to the same destinations as you would for 'general' knowledge - but truly, are you still 'searching' at that stage - or are you at the "knowledge stage" of purchasing, actually searching for "stuff" - I think it's the latter.

I think google already get this - the data we need to decide whether "general knowledge" is applicable to any "search" we do is different to the data we need to search for products - _froogle_ is google's early take of "horizontalizing" our quest for "stuff".

The debate here then should be, whether the quest for 'stuff' will ultimately be 'horizontal' or vertical? ebay has shown that the quest for 2nd-hand stuff is horizontal, but online customers of 'stuff' are yet to decide if buying 'new' stuff from amazon or froogle is best - or whether they'd rather go direct to the "manufacturer", who is capable of building a personal relationship with their customer and delivering personalized services.

A "horizontal" web implies that there's a (single, uber) middle-man. Interesting. One promise of the www was that it would remove the middle-man. Maybe the www needs to be both horizontal & vertical. I actually think that in our search for "stuff", we'll increasingly cut out the middle-man and go direct, vertical, to transact, after possibly using very horizontal services in the information phase.

tx for the debate

[Sean O'Rourke]

>Danny> "Vertical search is going to take over."

>Tom> "horizontal search of the whole accessible web will continue to be the starting point for finding almost everything including the structured search engines themselves."

I do not see these statements as mutually exclusive.

Vertical search, including structured search, can "take over" (grow considerably) within the context of horizontal search.

Of course, it can, and will, continue to grow outside the context of horizontal search, too.

That last statement was not entirely true. This is no "it" regarding vertical search.

There are two types of vertical/structured search tools, or perhaps two ends of a search tool spectrum: the ones that are used often, and the ones that are used seldom.

The tools that become part of people's everyday activities (travel, jobs, dating, etc.) will accumulate repeat visitors, who will bypass the horizontal search engines to reach them.

The tools that are used infrequently, such as a baby name search engine, will be critically dependent on horizontal search engines, even to be re-introduced to their repeat visitors.

Granted, the first group will always be dependant on the horizontal search engines for new visitors, plus the people who use horizontal search as their primary form of navigation.

And this whole post falls apart if you exclude structured search, but I have to assume Danny was including structured search when talking about vertical search.

Regardless of terminology, we are entering an exciting time with regard to structured search databases -- their number, their richness, their integration with each other, and their integration with horizontal search.

I will refrain from trying to make comparisons to TV. Any analogy between the Web and TV is guaranteed to be a sub-optimal analogy, regardless of the actual analogy.

[Faisal Premji]

Tom,

Your colleague's notions regarding vertical search are common these days, albeit incorrect. The error is apparent when one identifies the shift of 'semantic responsibility' from the user to the engine.

Simply put, the burden of semantic assistance shifting from user to engine will render vertical engines un-necessary on the front-end, but integral on the back-end.

Google's current state is all things to all topics. Unfortunately, this shotgun approach results in the onus being placed on the user to narrow the query - the 'semantic heavy lifting' is borne by the user, to 'tell' the engine what exactly he/she _means_, via detailed queries, boolean operators, etc..

Currently, a user would have a reason to go to a vertical search engine - half the semantic workload is already borne by the technology. The user and machine are on the same page, so to speak, from the outset.

However, google learns more daily about a) the relations of certain keywords or keyphrases to topics (already done on a rudimentary level, google can identify a zipcode, area code, UPS/Fedex tracking ID)

and b) the preferences and histotical actions of a specific user leading to a picture of their semantic world (java means coffee, not programming, since Mary is a computing neophyte)

these factors lead to the demise of the vertical search engine as a front-end interface - because the user has no need for the vertical's semantic assistance. google either has anticipated or already has a good idea of 'what we _mean_'

the value that a vertical provides will endure on the back-end (DB), not the front (UI).

[Robert Young]

My vote in this debate goes with Tom. Having said that, even if vertical search does gain significant traction, I wouldn't count Google out.

As we all know, Google's unique PageRank analyzes links to yield high precision, high recall search results. But if you look deeper to see why it works so well, you'll find it's due to the algorithm's ability to automatically *categorize* the web. The resulting multi-dimension directory is not only powerful for horizontal search, it can also be very effective for mining vertical categories. So with a few tweeks here and there, Google has the ability to be a player in both spaces.