Digging Deep

This has changed and as CIOs integrate more of the enterprise’s content under the search technology just the opposite problem is emerging: business analysts are now being overwhelmed by too much useful information. Productivity tools that work with traditional search engines are rapidly emerging as the next generation search technologies. Bob Carlson, CEO of Intelligenxia talks to BMUS about these new tools that enable users to work more productively and efficiently focusing on producing high quality and important business insights and not just reading the rich data returned by search.

What exactly are next generation technologies? Why is this different to the current market direction?

Next generation search is about delivering new productivity solutions to business analysts that leverage current investments in traditional search technologies. As organizations improve their search infrastructure they are actually decreasing their business analysts’ productivity. This non-intuitive outcome is because analysts are spending more time than ever reading the useful results from their search rather than performing their primary task of producing business insight for senior management.

Next generation search is about analytics and analysis and not just access and reading. Next generation search augments search technology to create an integrated business analytic capability for both unstructured and structured information.

What is undisputed is that search has become a strategic part of most company’s IT infrastructure. Search tools are widely available to employees and have encouraged the CIOs to integrate more and more of the enterprise‘s content under the search index -- making it accessible to those that need it. In parallel, IT vendors are investing hundreds of millions of dollars in R&D to improve search. What is interesting is that the definition of improved search has changed from “finding the needle in the haystack” to broader value of “returning only relevant documents to the user’s search request”. While the hundreds of millions of dollars of R&D will improve search we will continue to suffer the negative impact to the business analyst’s productivity, as they are required to read all this useful information.

For next generation search to be successful the solutions must provide improved productivity while leveraging the investment already made in enterprise search.

Why can’t continued investment in search provide this capability?

The large search companies like Verity, Autonomy, Google, IBM and Microsoft are definitely investing heavily in R&D to improve search. The reason this won’t solve the productivity problems is that these new capabilities are focused at providing higher quality search results and not focused on what users need to do with these results.

This extraordinary investment should ensure that your search result will decrease from millions to thousands of very useful and relevant returns. But even so, at the end of the search effort analyst will still need to read these thousands of useful documents in order to complete their job. We don’t think you can provide better business results by requiring someone to read thousands of documents. What is required, and what we provide, is a companion technology that allows the business analysts to work with and analyze these documents in an efficient and intuitive way so they can provide meaningful insight back to management.

Unstructured data typically accounts for 85 percent of an organization’s knowledge store. What are the benefits of analyzing this data?

Unstructured data has really become the critical link in decision making. It’s the email, white papers, presentations and documents that flow from the line of business owner to the brand owner to the manufacturing owner to the financial owner that support the decision making process. Accessibility to this critical corporate asset is providing unprecedented clarity into business processes and problems and is why companies are investing in technologies like search, document management, portals, and email archiving.

What business leaders are beginning to recognize is that while accessing unstructured information is critical, if they could better understand the content and interrelationship held within this information then they could make faster, better decisions.

What we are beginning to see in small, but crucial first steps, is that companies are spending time and money looking at, for instance, employee feedback surveys to identify overlooked opportunities for improving process efficiencies or delivering better customer service. This feedback is provided to the organization as open-ended surveys, email, “jam sessions,” or chat rooms comments -- all of which are unstructured data. Now that search has made this information visible, innovative business leaders are looking at ways to leverage this access to make better decisions.

What solutions are now available to do this and how do they convert that data into something more useful?

As soon as the Netscape browser was made available in1995, people began to imagine the business value of all this unstructured information. Two different approaches were independently launched to accomplish this: search and text analytics. The text analytic approach had similarities to search but it was focused on creating rich Metadata for each of the documents ingested by the technology. The vision was that this would provide business users with more sophisticated access to solve business problems. Search is a more straightforward and efficient approach focusing on simply providing access to data and not trying to support sophisticated, complex business queries. These two started in parallel but search has established itself within the enterprise while text analytic adoption has been a disappointment.

Looking forward I believe a whole new generation of companies will develop technologies to sit on top of the search environment. These solutions will allow customers to focus their incremental investment on extending the search value proposition and not “ripping and replacing” the search infrastructure with a new text analytic infrastructure. This is the big change and frankly I believe it is going to allow the marketplace to grow faster and deliver more value at less cost to our customers.

There is much talk of solutions such as text mining. What problems could an organization avoid by investing in solutions such as these?

I think this is one of those questions driven by a vision that we will never achieve. One of the visions of the text mining companies is to develop technology that can find answers or insight in the data that the analyst is not even looking for. Millions of dollars of VC funding has been invested to develop a technology that would deliver “Ah ha’s!” to management. But, that’s not how it works. Serendipity is neither a sustainable value proposition nor a capability that technology can deliver. What I believe, and what we are trying to do with our technology, is move the discussion from “our algorithm is better than their algorithm" to “we can help your analysts shift their time from reading documents to performing the critical analysis that creates value for the company and the shareholders.” I think this is a crucial change that you’ll see more of in the marketplace.

What are some of the restrictions of text mining and how can we overcome them?

A detriment to text mining is the perception that it is a “black box technology” making it really hard for business people to understand how it works. Companies have traditionally competed on the premise that “my algorithm is better than their algorithm” and seem to have made the technology intentionally difficult to implement and support. The way to encourage the adoption of these new approaches is for open, standard architectures to emerge. This is actually happening now. As we progress, companies will be able to integrate the appropriate capabilities themselves or work with their trusted services partner to use the technology to solve business challenges.

What would your top tips for generating rich data be?

I believe the marketplace has demonstrated that a solution that tries to anticipate every question your business analysts might have is a failed strategy. First, as you try to build the next generation of search, you should resist the temptation to drive more and more complexity into your corporate taxonomy. You should create a taxonomy reflecting the company’s view of the market, customers, products, and competitors but allow the personalization of this data to occur close to the business analyst at the moment in time when it can provide value. The second tip is to look at corporate taxonomies the way you look at any corporate database: it’s there to support a primary capability and should be optimized for that capability. For the corporate taxonomy it should support the best possible search strategies with fastest possible response times.

Next, we are going to learn a lot more about the types of taxonomies that support business processes and we should expect, just like we experienced in the database world, standard taxonomies to evolve in the future. Just as SAP has standardized the corporate data model for functions like finance, a similar standard taxonomy for an industry will emerge.

Finally, the analyst must be in control of the structuring the data required to complete the task. New solutions must avoid rigorous internal processes aimed at structuring the data.

Some might say installing a solution is easy; the hard part is getting meaningful results from a process that depends on the skill and knowledge of the person using the software. Would you agree with this statement? Is there a big skills problem in this area?

I definitely agree with the statement, but I don’t think it’s primarily a skills or lack of skills issue. I believe the technologies delivered up to this point have focused on the algorithms and not on ease of use. This is an area where we have spent a tremendous amount of investment to create an easy-to-use, intuitive user interface. Our experience is that this approach has allowed our customers to quickly come up to speed and become productive, with very little training, using the tool.

Our software is a productivity tool for business analysts and for us to be successful we must deliver a product that is both easy-to-use and intuitive allowing the business analysts to easily incorporate it into their current workflow. Our approach provides the business analyst with control over how the technology will be used and eliminates the need to re-educate millions of analysts on how they should do their job. We see the business analysts as the experts and these experts will use our software if we provide them unique, easy-to-use capabilities that help them increase their value to their organization.

How much has the demand for unstructured data analytics solutions grown in recent years? What will drive the market forward in the future?

The demand has always been there. Up to this point the technology complexity and lack of an enterprise search infrastructure were barriers to effectively delivering software that could satisfy the demand. With search we finally have a platform to build upon. We see demand for our product everywhere business analysts try to use search as a research tool. In other words, search is being used for a number of things that the original inventors never imagined. Our technology is all about helping business analysts who are now overwhelmed by have access to too much useful information. If they can become the masters of this information then their value to the organization will be greatly enhanced.