The Future of Search

The search and discovery software market grew to $1.8 billion in 2007 at a growth rate of 28%, outpacing the software market as a whole. Sue Feldman believes that it will continue to do so in 2008, despite a slowing economy. BM looks at what is driving this increase.

Interfaces and interaction will change radically

“We will see that technology will understand the importance of a question, or engage in a conversation with the user, so very much like two humans might in which you would ask a question and remove the ambiguity of that question.”
-Sue Feldman, Research VP for IDC

The value of search as a true enterprise platform has been touted for years, yet remarkably few organizations have seriously embraced the opportunity, until now. Sue Feldman, Research Vice President for IDC, believes that this is because IT has gotten used to implementing transaction-based applications. “It is only recently that enterprises have begun to understand that they have a valuable resource in their unstructured information,” says Feldman.

The main driver for adoption has been that understanding has started to spread. “Those of us who have been in this field for many years have always looked at the technologies we deal with as being completely different from database technologies,” explains Feldman. “However, there hasn’t been much headway up until the last couple of years.” Feldman goes on to explain that what has changed in the past couple of years is that compliance and risk management have brought this awareness beyond IT and up to the boardroom.

Organizations realized that they needed a greater degree of control and insight into their unstructured data and best practices have since evolved extremely quickly over the past couple of years, so that they have become better compliant with regulations. “We are wallowing in a sea of information,” says Feldman. “The company information has to be pinned down and managed better and if that doesn’t happen then companies are severely at risk for not complying with regulations and putting themselves in jeopardy.”

Growth

Given the sensitive nature of the data involved it seems the market can only get bigger. In 2007, the sector grew 28% and according to Feldman’s Worldwide Search and Discovery Technologies Forecast, 2007-2012 report, the market is predicted to grow 12% year on year to a total of almost $3.2 billion by 2012. While Feldman believes that the software market as a whole is going to grow by around 4 to 5%, she claims the search and discovery market is interesting for a couple of reasons. Firstly, because it is a new foundational technology and Feldman likens it to the “beginning of the database era”. And secondly, that the technology is important in a various number of ways, which makes people spend on it when they wouldn’t spend on other things.

“Email looms as a real threat to organizations because it is so poorly managed and yet so vital to how information workers glue together all of their applications today, but it is also a tremendous opportunity to become a valuable source of information about the current business of the organization,” explains Feldman.

Technology

Knitting together all these disparate technology systems, repositories and databases is hugely challenging, but a number of technologies are being built into this in order to unify access to information and pull together data and related content in one place. Feldman cites a number of technologies that are allowing organizations to use their data to converge search and discovery with business intelligence. “I can’t emphasize enough the importance of pulling together both data and content into one single access point, and that interface is going to become increasingly important because that’s what determines the efficiency with which people can accomplish their tasks,” says Feldman. 

Feldman predicts that in the long-term technologies will be able to understand language in some way. “We will see that technology will understand the importance of a question, or engage in a conversation with the user, so very much like two humans might in which you would ask a question and remove the ambiguity of that question,” says Feldman. “That’s where I think that interfaces and interaction design are going to change radically.”

While personalized, contextualized results for all knowledge workers will be produced, Feldman is keen to highlight that it is the conversations themselves that are key. “We want answers and we want a conversation to get us those answers.” In this respect, it is more about the journey than the destination. While it will be a gradual evolution towards conversational search it has started already. Feldman believes that looking at the latest crop of web search engines is a good indicator of what is going to happen as consumer adoption feeds enterprise adoption and vice versa. “None of what I would consider to be this generation of search engines returns just a list of documents. The standard today is already indicated answers and clusters of topics related to those answers, so that you can drill down,” says Feldman.

Future trends

Feldman also believes that there is a trend towards hybrid applications in which you see search, plus workflow, plus collaboration, plus sort domain or industry knowledge incorporated into a single work environment that is geared towards a particular task. “Search is becoming a component of this in order to find information within the natural workflow of a person’s day. This is what we are going to start seeing in the future and the conversational systems are a piece of that,” explains Feldman.

Hybrid applications are already underway and Feldman believes that the Lexus Nexus interact is a good example in which it is possible to see an environment that was created to support marketing officers in large legal firms. “EMC has spent months studying how information workers work to create environments that include collaboration search and workflow and you are going to see more and more of these,” says Feldman.

As the search and discovery market continues to grow, incorporating increasing amounts of technology and companies become increasingly dependant on organizing, finding and using information, doing it quickly and thoroughly is becoming crucial. Feldman believes that as technology develops the market will undergo huge changes. “I would bet that in the next five to 10 years there will be such a difference in how we are able to interact with computers that we will look back at this decade as being the Stone Age,” says Feldman.

Extra info…

Search and discovery forecast
The search and discovery market in 2007, and in the first half of 2008 continues to outpace the software market as a whole. While the pure play search software market continues to consolidate, with mergers and acquisitions still rampant, search technologies have also begun to penetrate traditional data-centric enterprise and consumer applications such as CRM, recommendation engines or ad matching. IDC believes that:

1. The search market will continue to fragment into three tiers: OEM, solutions and platforms, with solutions that address a specific task or problem growing at a faster rate than the other two market segments

2. Text analytics vendors as a whole will continue to see even faster growth than the search market. Hot areas include sentiment extraction, eDiscovery, geo-location and language modules

3. Although this market has continued its rapid growth in the first half of 2008, economic indicators for IT spending are bleak. For that reason, IDC expects slower growth in the search market, with no more than 17% growth forecast for 2008 and 12% for 2009. This is down from the 28% growth we saw in 2007.

4. As search and text analytics become technology features in larger software applications, we can expect the current spate of technology-driven mergers and acquisitions to continue. At the same time, new search and discovery technologies vendors continue to emerge, broadening choices and adding to market confusion.

[Source: Worldwide Search and Discovery Technologies Forecast, 2007-2012, IDC]