Cuil, Microsoft, and the future of search
There are three parts of the searchable Web ecosystem: Publishers, Indexers, and Searchers. This ecosystem has evolved three areas of search optimization: Publishers adjust the quality and relevance of their content; Indexers adjust the quantity and metrics of the algorithms and databases; Searchers adjust their queries. At a meta level, the searchable Web ecosystem forms a symbiosis: Publishers need Indexers to make their data available to Searchers; Indexers need Publishers to provide Searchers value; Searchers need Indexers to find Publishers. Everyone needs everyone else.
When the Cuil search service launched on July 28, 2008, a great many naive people wrote some pretty scathing remarks about the service. Cuil’s servers crashed, query operators that people had become used to didn’t work, their press release made some pretty bold claims, and a lot of us noticed funny pictures besides otherwise relevant search results (in some cases the relevance was lacking).
It has been a very, very long time since people have had the opportunity to view natural search results. The only optimization that went into Cuil’s results on July 28 was the algorithmic optimization that its engineers employed. That is, in the micro-ecosystem of Cuil’s searchable Web environment, only Cuil was able to optimize the search results.
That mostly remains true today, a little more than a month later, because Cuil has yet to introduce any query operators for Searchers and because Publishers have only made limited attempts to optimize Cuil’s search results. You can contact Cuil directly (for now) and ask them to fix something that is wrong. That window of access will one day close.
Some very smart people have waited until the pack has passed judgement before taking a more thoughtful look at Cuil. But it’s still only a month after the search engine was launched. To ask that Cuil become a “Google-killer” within one month is equivalent to asking Monaco to become a world power within one year.
Both Monaco and Cuil have potential for growth. Monaco can, at best, seize some acreage from the sea through proven land reclamation technologies. I don’t think they’ll ever acquire the resources to become a world power.
Cuil, on the other hand, has a much larger space into which it may expand. They can increase their resources, enhance their technology, and ultimately show people that they are here to stay. It is way too soon for people to care about how much traffic Cuil receives after the launch.
Using number of searches performed as a metric for search market share is ridiculous anyway. Who is running rank checking software against Cuil? People hammer Google with bogus queries all day long, so gauging market share by queries is equivalent to judging a horse race by how pretty the saddles are.
Let’s take a little look at search history:
The two leading search providers in 1994 were Yahoo! and Lycos. In 1995, Excite, Infoseek, and Altavista joined their ranks.
During these early years the search market share was evaluated by number of visitors. In August 1997, according to the paper I just cited, the search market looked like this:
| Search Market Share in August 1997, Measured by Unique Visitors | ||
| Yahoo! | 14,800,000 | |
| Infoseek | 7,900,000 | |
| Excite | 7,600,000 | |
| Lycos | 4,900,000 | |
| Altavista | 4,700,000 | |
| Webcrawler | 3,200,000 | |
This data was published by MediaMetrix (now comScore), whose Press Release archive includes all Media Metrix data that was made publicly available in the 1990s. By 1999, the market had changed:
| Search Market Share in August 1999, Measured by Unique Visitors | ||
| Yahoo! | 33,00,000 | |
| Infoseek1 | 18,500,000 | |
| Excite | 14,900,000 | |
| Lycos | 14,100,000 | |
| Altavista | 9,200,000 | |
| About | 8,600,000 | |
| Looksmart | 8,500,000 | |
| Snap2 | 8,300,000 | |
| Hotbot3 | 7,200,000 | |
| GoTo4 | 7,100,000 | |
| AskJeeves5 | 4,000,000 | |
Notes:
- Infoseek was purchased by Disney and rebranded as Go.com.
- Snap was purchased by NBC and rebranded as NBCi. Today’s Snap is a different search service.
- Hotbot was one of more than 30 search services using Inktomi technology.
- GoTo introduced paid search listings, combined with Inktomi results, and rebranded itself as Overture.
- AskJeeves acquired Teoma and rebranded itself to Ask
Digital Equipment Corp. launched Altavista in December 1995. In 1996 Yahoo! began supplementing its directory search results with Altavista’s algorithmic Web search results. From this point forward, whichever search engine was partnered with Yahoo! was considered to be the top search engine in the industry. In 1998 Yahoo! dropped Altavista to partner with Inktomi.
Compaq acquired Digital in 1998 and in 1999 reconditioned Altavista to be a Web portal. This was the beginning of the end for one of the most innovative search resources on the Web. Compaq sold a majority stake in Altavista to CMGI, which in turn sold Altavista to Overture in 2003. A few months later Yahoo! acquired both Overture and the Altavista technology, which was at that time the best in algorithmic search. Yahoo! abandoned the Altavista technology in favor of the Inktomi technology Yahoo! had acquired in 2002.
Inktomi’s technology had some advantages over Altavista’s, but Altavista had the better search results, especially toward the end. Inktomi powered Yahoo!’s Web search results from about 1998 until 2001, when Yahoo! dropped Inktomi for Google.
In 2002 Yahoo! moved its Web search results in front of its directory search results and then launched its own crawler-based search in 2003, eventually dropping Google.
There are two lessons to be learned from this timeline: First, Yahoo! was the undisputed kingmaker while it was an independent directory and then portal. Without Yahoo!, it’s unlikely Google would have become the top-ranked search engine. Second, superior search technologies don’t always win out — partially because search providers have to make some hard choices.
In 2002, Altavista actually had better search results than both Google and Inktomi (through the experimental Altavista Raging search tool). So why did Altavista fail? The instability of Altavista’s resources ultimately led to its failure. Altavista did make money for a while but that money was driven by banner advertising revenues which collapsed in the dot-com meltdown of 2000-2001 and by the Yahoo! partnership. The loss of revenues forced Altavista’s owners to get out of the search business. Altavista’s technical team was broken up and pushed down the list of priorities by CMGI.
In retrospect, if Altavista could have found a way to monetize search like Goto/Overture and Google had, they might have pulled through and survived. But when people talk about the superior technology that Inktomi and Google used to displace Altavista, they are referring to more than just search results.
One of Altavista’s biggest problems was load balancing. They just never seemed to have the server power to get the job done. You could run the same query twice in a row on Altavista and see completely different results because if the server was stressed it would just stop processing queries and hand back whatever results it could. People were ultimately disillusioned by the inconsistent quality of the results.
Google and Inktomi both solved the scalability problem, which ultimately led to what we could call today’s Scalability Ceiling. That is, in order to compete with the four major search engines (Ask, Google, Live, and Yahoo!) you need a type of resource that is very expensive: a data center. In fact, the major search engines use multiple data centers.
Ask, Google, Microsoft, and Yahoo! have also solved another problem: internationalization. In the 1990s American search providers really did not have to worry about overseas Web sites. There were so few of them they could easily fit in with American Web content. But that was the 20th Century Internet. In the 21st Century major search providers are expected to provide regional-specific search interfaces (and language-specific search interfaces, too).
Good search ain’t cheap, and these are all good search services. To compete with them, any newcomer has to solve these issues (and others). People have become accustomed to robust features: they want Local Search and Video Search along with Web Search. Universal Search satisfies those demands. People also want to be able to refine their queries.
New search services like Cuil cannot afford to start out with 50 data centers, tons of query functions, and hordes of search verticals. They have to build a solid crawling, indexing, and query resolution technology first.
Furthermore, there is no longer a kingmaker in the business. Google has effectively dominated search since 2002 because no other two search services have been able to join forces against Google. Yahoo!’s consolidation of search services Inktomi, Goto/Overture, and Altavista effectively removed all the potential Google-killers from the competition, although the consolidation resulted in great part from the dot-com meltdown.
That is, when the Internet bubble burst, Inktomi’s several dozen Web search partners began vanishing. Without those revenues Inktomi had to rely upon Enterprise Search, but even that market suffered from slowdown. Yahoo!, on the other hand, saw its dominant position eroding in part because of its own dithering. It didn’t know if it wanted to be a directory, a portal, or a search engine.
Other directories were collapsing and reinventing themselves during the years 2000-2002. Looksmart shrank to a shadow of its former self. Disney shut down its GoGuides directory and abandoned search altogether. NBC decided it could not afford to turn NBCi into a profitable business (despite the fact it had one of the best directories on the Web). Excite went through a disastrous merger. Lycos vanished into Terra-Lycos. Search was an unattractive proposition for all the former leaders of search.
In effect, the search industry collapsed and Google was right there, waiting to fill the void not only with an adequate search technology that was scalable, it found a way to monetize search efficiently. Without AdWords, Google probably would have failed eventually as well. You cannot keep burning the candles if you don’t have the money to buy more candles.
So new search services like Cuil, Hakia, Mahalo, Me.Dium, and Wikia not only don’t have a Kingmaker to court, they have no void to fill.
So where does this leave us? Believe it or not, it leaves us with our hands full of opportunity. Those search optimizers who find efficient ways to optimize for these younger search services will be ahead of the field. Google’s chances of remaining on top of the industry are declining each month as Microsoft gains ground.
Steve Ballmer clearly hoped to forge an alliance with Yahoo! that would have repeated history: I think he wanted Yahoo! to play the role of Kingmaker once more, annointing Microsoft’s Live Search as the next market dominant search engine. I never believed that strategy would work because there was too much friction between Yahoo!’s board and Microsoft. One of the brands would have to be sacrificed.
Nonetheless, as Microsoft’s visitor share grows at near astronomical paces (whereas Google’s growth is barely limping along), Microsoft is definitely in a position to overtake Google and seriously challenge it for market leadership within the next year.
The repercussions of such a struggle cannot be estimated. Google and Microsoft could both come out stronger, enjoying joint leadership in the market the way Yahoo! and Lycos did in the early years. Or else a new player could take advantage of their intense rivalry to outmaneuver them. Yandex, for example, just went public this year and is starting to index the English-speaking Web. Yandex definitely has more resources than Cuil to work with.
So what if Cuil and Yandex were to join forces? Each would bring strengths to the table. I don’t know how good Yandex’s technology is, but Cuil definitely has a huge technological advantage over Google, Microsoft, and everyone else in one critical area of search technology: Cuil knows how to determine relevance like nothing we’ve ever seen before.
People may be disappointed in Cuil’s performance overall, but Cuil doesn’t actually need SEO help to serve relevant results. Google, on the other hand, flounders like a fish out of water if people don’t optimize its query space with links and on-page emphasis. Google search results for unoptimized queries are often very irrelevant, and not for lack of content to index. Yahoo!, Microsoft, Ask, and Cuil all show content for many non-competitive queries that is more relevant than Google’s search results.
That disparity in quality and relevance is Google’s Achilles heel. Google’s overdependence upon link anchor text and PageRank does, actually, make it vulnerable to both manipulation and competition from smaller, less well-funded search engines. All Cuil has to do is show people that it can expand its functionality, improve the quality of its results where they need to improve, and capture a few key endorsements and it will be in a position to negotiate for the resources it needs to challenge the major search engines.
But in order to mount a serious threat to Google, every small search engine today will first have to climb over the broken backs of Ask, Yahoo!, and Microsoft. You cannot challenge the leader without first beating the crap out of his bully boys. Google runs with a very small, very tight crowd. Gaining admission to that crowd is the first order of business.
Instead of scorning these smaller services, search optimizers should be looking for ways to hedge their bets by optimizing the results of those services. Each of today’s new generation of search providers brings something unique to the table. None of them may succeed at breaking into the top ranks of search, but they could all foreshadow new technologies that emerge and rise to the top in the next 5-10 years.
If I had to pick one, only one of these search engines today, my choice would be Cuil. Cuil offers us a unique interface that is informative and much easier to use. A lot of people have complained about that interface but transitions in interface paradigms have occurred before. People will grow used to a useful tool. What they won’t do, however, is fall in love with a bare-bones tool.
Cuil has to expand its capabilities in order to compete. It has to be able to handle complex queries, and it has to be able to scale without sacrificing the quality of its results. Because Cuil is focusing more on the page content than other factors, most SEOs are poorly equipped to compete in a Cuil environment.
You’ll do better to look at Cuil more often than to look down your nose at it. Or, if you cannot fall in love with Cuil, then you’d better pick another dance partner from one of the other new contenders. There will come a day when people are looking for the guru who knows how the new search engine works. That won’t be you if you cannot take your eyes off of Google.
And if Microsoft continues to build real market share at its current pace, that day will come much sooner than you expect.
feeds.feedburner.com
published @ September 2, 2008