The search continues

Enterprises are increasingly focused on the problem of access to and discovery of actionable information, according to Bradley Allen, founder and CTO of Siderean.

The amount of digital information in the world is growing on the order of several exabytes per year. This tsunami of data and content comes from many different sources. Most enterprises are struggling with uniting internal data silos. Add to this the convergence of data from outside the firewall and you have a serious information access problem. Any and all of this information has the potential to be crucial to the business processes and decision making of a given enterprise. Users struggle with discovery of and repeatable access to relevant information that they need to make informed decisions; i.e., information that is actionable.

The business impact of this flood of information is heightened by the fact that enterprises can no longer assume that they completely control either the flow or content of the information about their products and services to their customers and partners. The effort of understanding and managing the flow of this information is significant, and access and discovery across all these different sources of information is increasingly a major focus of the enterprise. This is the main reason why we are seeing an uptake in search solutions today.

Search is insufficient to solve this problem
Enterprise search and discovery as a market has become a billion-dollar market over the last few years, by IDC’s estimates growing on the order of 30 percent in 2006. Unfortunately, and in contrast to the tremendous success that search has had in becoming a pervasive user experience over the last decade and a half, search as it is traditionally conceived is insufficient to address the problem described above.

Traditional search involves free-text indexing. Free-text indexing takes the contextual cues (or to use the technical term, “metadata”) out of a document and throws it all away, leaving you with just a “bag of words.” When this happens, much of the value in the information that allows people to understand were it came from, how recent or reliable it is, how it relates to other, perhaps critical, pieces of information is lost forever.

This creates a situation where the user has to make up for the loss of information by being clever about how to craft keyword queries. Either too much irrelevant information is returned, or too little, and users are then confronted with the task of deciding how to proceed, with little or no useful advice from the system. In no way does the system give the user an indication of what information is available to them. Users need a way in which they can master discovery of actionable information, and search as it is presently implemented does not achieve this end.

The future of information access and discovery is navigation
One particular area that has the potential to emerge as an alternative to search that directly addresses this problem is navigation. Navigation, as pioneered by companies like Siderean and Endeca, allows enterprises to give their users a unified view of what’s available, allowing them to zero in on what’s relevant, find and follow relationships, and share discoveries with others. This is a huge leap forward in making information actionable, and subsumes search as one of the information-finding tools that it provides to the user.

By using navigation based on metadata to provide context, users can orient themselves and find their way reliably through vast information spaces, without having to be initially aware of the vocabulary being used in the documents. The focus of effort moves from having to think of what keywords to type to one of selecting from a set of alternatives presented by the system based on the current context of the user’s interaction with the system. It is a matter of recognition versus recall. This contextual framework gives users the ability to hone in on information from many different directions, increasing their chances of success in finding a specific piece of actionable information.

Faceted navigation provides users with better scope than search, but only along fixed views predetermined by marketing or IT. Relational navigation, pioneered by Siderean, additionally provides user with the ability to discover relationships between different pieces of information that may have been hidden in the mass of results obtained through keyword search. Instead of simply searching for documents that match keywords, the user can explore and comprehend the relationships between people, organizations, events, topics, and the documents that bind them together, leading to deeper insight and better decision-making.

This approach to information access and discovery has demonstrated reproducible results in improving revenue of e-commerce sites by allowing users to more effectively navigate product catalogs, and promises productivity enhancements in the use of relational navigation as applied to knowledge management and enterprise information access. This success and increasing adoption is a significant trend and a big part of the future of search in the enterprise; as traditional search becomes increasingly commoditized by open sources solutions such as Lucene, we expect that relational navigation will emerge as the central organizing principle and platform for the information access and discovery market over the next five years.