Search and beyond in the digital marketplace

About web search versus enterprise search, the realities of ‘The Google Effect’ and where the evolving digital marketplace is headed next.

As the transformative digital marketplace has disrupted traditional business models across the enterprise, both threats and opportunities have emerged for markets and vendors alike. Amid today’s thriving digital media ecosystem, good search has become a critical component necessary for enterprises to excel in an information-rich age. While expectations for enterprise search have been erroneously based on web search experiences, a number of standout vendors have emerged to help build robust enterprise search platforms.

Google, who can more or less be dubbed the architect of the digital ecosystem, has infused change across almost every model and market possible. Here, Business Management hears from IDC’s Susan Feldman (who coined the phrase ‘The Google Effect’) on how Google has not only catapulted search into a prime business tool, but has spurred innovation throughout today’s digital marketplace.

BM. Can you articulate how an enterprise search engine requires different capabilities than web search engines?
SF. Consumer search and business search have different requirements; they also search different kinds of materials. If you’re a consumer, you’re searching for an answer, not every possible bit of information that exists about a subject. A relevance-ranked list of results very often solves your problem. But businesses require both accuracy and comprehensiveness. They must be sure that they have all the information, but they must also weed out the extraneous information that takes too long to wade through.

Another difference between web and business search is that the web covers every topic imaginable. This means that the differences among documents are easier to determine. Within a business environment, you can expect that there are many documents that are very similar – in fact, it’s likely that only one or two sentences, plus the date of publication, are all that differentiate one document from another. However, that difference can be crucial: it can mean the difference between this year’s contract and last year’s.

If I go to a clustering engine on the web and I type in Boston, it’s almost certainly going to return information that would help me be a tourist in Boston, such as Boston hotels, sporting events, concerts and so on. But I don’t need the names of every restaurant or the details of every event in order to find out what to see in Boston. Within the enterprise, it’s a different story. Let’s take IDC for example. If you typed in services-oriented architecture (SOA), that term will probably appear in the writings of all of our 800 analysts – meaning you’ve got thousands of documents that mention SOA somewhere. If you want the definitive document about SOA, you’d better be able to tune the relevance to return the best document, or you’ll end up with too much to read that is of little relevance. So within the business, we first of all need to find the most relevant information and get rid of the stuff that’s peripheral. We need to make sure that we’ve got the right information for making decisions and we need to find that information fast.

Security requirements are also a huge differentiator for enterprise search in terms of how to ensure that access to information such as financial records and employee benefits will be secure. The right people need quick, easy access to this information, but at the same time, those who don’t have permission to see all or part of it need to be prevented from even finding those documents, let alone reading them.

Business search requires subtlety and flexibility, and that makes a difference to the kind of technology you need. An enterprise information platform must also support a broad spectrum of information-seeking tasks, as information finding within the enterprise is almost always part of a larger process. A big research project means searching across multiple collections, both within and outside the enterprise. You are looking for a comprehensive set of documents to get you started, and then you need to read them and analyze them before you distill what you have learned. That’s different from locating a single document within a database or a slide deck. Suppose that I don’t even know how to ask a direct question; I will then need clues to help me find what I am looking for. I’ll need to browse through a directory or taxonomy to find the right general area. I may never type in a question at all. Questions such as “Give me a graph or a chart that visualizes sales trends over the last five years” or “What products are our customers complaining about?” and “What are the causes of those complaints?” are valid questions that you would expect to be able to answer with an information system within the organization, and each of them requires a different combination of technologies.

Enterprise search software tends to be much more advanced because it must support multiple tasks. It requires very advanced security features and subtlety of the relevance ranking. And search is just the beginning of an enterprise information access platform. These days, you would expect, at a minimum, that these platforms would support access to both data and content; that they would include a search engine, plus some sort of categorization; and probably some text and data analytics. They would need robust security, administrative tools, information visualization and reporting capabilities for both monitoring the system and for analysis.

The problem is that these differences are often not apparent. Confusion arises when people don’t understand how the use of information is different within the organization from how they use search on the web. They expect that they can satisfy their information needs with just a search engine because that’s what they do personally. They don’t realize that it’s much more complex once they get within the organization; slapping a search box on top of multiple sources of information, and expecting that the relevance ranking is going to be automatically good and you’ll get everything you need, is a false expectation. You need much more than that in order to satisfy the requirements for information-finding within the enterprise.

BM. How do you think improved enterprise search capabilities can impact overall business performance?

SF. It’s easiest to prove the value of improved search in customer-facing applications. For instance, if you’ve got an e-commerce site, every time you connect somebody directly to the product that he’s looking for, the chances of making a sale are much greater. On the other hand, a poor search loses your customers. So hundreds of thousands of dollars in extra sales is common with improved search on an e-commerce site. Similarly, with online self-help applications, every time you answer a customer’s question you save money. Being able to respond accurately and satisfactorily to a question that a customer may have and doing it automatically racks up hundreds of thousands in savings every month.

It’s harder to quantify the value of a good information access platform within the enterprise. We know that people spend at least a quarter of their time looking for information every week (our data shows that on average they spend about nine-and-a-half hours per week looking for information). We also know that somewhere between 30-50 percent of that time is wasted because they don’t find the information that they need and then they spend more time recreating the research or data that they need because they can’t find it. So while the savings within the organization are harder to quantify, they may be even greater just in terms of time saved. And it is even more difficult to figure out the value of making better decisions based on more complete information. I know of numerous examples of disasters that have happened because people have not found the right information or have acted upon incomplete information, made the wrong decision and that has had a disastrous impact on the organization.

BM. Who would you say are some of the leading enterprise search vendors that can help extend search capabilities, and how do their offerings set them apart?

SF. In terms of pure-play search vendors, Autonomy, FAST and Endeca would be the top three in order by size of revenue in 2005. To that you have to add IBM – its new OmniFind search engine – and the Google Search Appliance. So those would be the top five based purely on revenue. If you want to talk about technology, Google doesn’t fit in this category yet but you can expect that they will; right now, Google offers a search engine, which is only a piece of an information access platform. Autonomy, Endeca and FAST all have the capability to federate to numerous sources of information, categorize it, visualize it and combine data and content sources, while applying document and sub-document level security to the results. Administrative tools let you tweak the relevance ranking in order to make it pertinent to your organization. They also do some sort of text analytics, analyzing patterns within the text or extracting concepts and their relationships to each other as well as people, places and things. They are extremely flexible platforms.

BM. Taking a look at the web 2.0 culture, what does the term ‘Google Effect’ mean to you and how do you feel it has affected the workplace?

SF. I started this research at IDC a year ago. We called it ‘The Google Effect’ because the word Google kept cropping up in technology market discussions across IDC – hardware, software, software as a service, services and telecomm analysts were all mentioning Google and the effect it was having on their traditional IT markets.

The Google Effect is almost like the butterfly effect in chaos theory: Google acts in a small way and causes a ripple in the industry. For example, publishing is threatened because of copyright issues with Google Scholar; telecomm is concerned because of Google’s entry into the wireless world; eBay finds Google Checkout worrisome because it competes with PayPal; Google’s powerful data centers have software vendors concerned about their delivering software as a service (SaaS). The Google Search Appliance has disrupted the search market not only because it’s much cheaper than an enterprise search platform, but also because it has confused the enterprise market. It encourages the consumerization of enterprise software. By this I mean that the simplicity and attractiveness of consumer applications is influencing how the enterprise considers the best way to deliver enterprise applications. We’re starting to see more concern about interface and being able to interact easily with applications without a great deal of training. I could go on, but The Google Effect really relates to the real and perceived effect of Google’s moves (or the threat of moves) into a variety of industries, and how the IT industry in particular is reacting to this.

BM. How do you think the evolving digital marketplace has affected and will affect traditional business models moving forward?

SF. That’s our research topic for 2007. If you think about the digital marketplace as a sort of parallel economy, you realize that most businesses need to have a stake in it. Several things are happening now: first of all, the long tail effect on marketing, as well as on what influences purchasing both online and offline. For example, small businesses suddenly can have a global presence without having to invest a huge amount in letting people know that they’re around. As Thomas Friedman says, the world has flattened and this digital marketplace allows us to do business globally, quickly and efficiently. Purchasing is another example: being able to purchase online as an enterprise as well as a consumer is big business now. The digital marketplace is being driven by a new kind of business model – which is advertising – and online advertising last year was about 11 billion dollars. Goodness knows what it’s going to be this year. That’s another source of income that has enabled brick and mortar companies to do business globally, to extend their reach as well as to increase their presence and brand awareness.

BM. What do you see as new or emerging trends in the digital media market looking ahead to the next 18 months?

SF. We expect that enterprises, and businesses of any size, will begin to demand the software and hardware infrastructure that will support doing business online. We’ll see the arrival of extensive tools and applications that large enterprises will use to create digital marketplace platforms. This will be followed by scaled down versions – a digital marketplace in a box. These infrastructures will combine search, web analytics, transactions, storage, download management and digital rights management (depending on what kind of business you’re in) into a logical infrastructure. People will be able to plug it in and become a node on the digital marketplace. It’s going to take time to develop that, but I think the notion is going to emerge this year.

Secondly, the importance of communities can’t be overrated, both in the consumer space and within the enterprise. How do you use social networking in order to create effective partner communities? Our research has shown that creating an effective partner network can very often spur exponential growth in business not only for you but for your partners as well. So we’re going to see more and more emphasis on how communities affect opinions and how you run your business. People will be pulled together into collaborative communities across the world. We’re also going to see lots of combinatorial innovation – mashups using standardized modules in surprising ways – like maps combined with local rental information or restaurant reviews. Businesses will be launched based on combinatorial innovation. There’s so much going on that it’s just kind of astonishing.