Business Intelligence, Document Management and Enterprise Search

Take a moment to let that sink in.

• 80% of your information isn’t available to your report writers.
• Your best analysts only “slice and dice” 20% of the information that’s available to them.
• Decisions based on “the most and best data” typically only take into account one-fifth of what they should.

What can you do about it?

Enterprise search software appears to solve part of the problem, because it incorporates unstructured documents (e.g., word processing documents, spreadsheets, PDF documents, and email) into the same architecture as classic reports. A simple search box brings a user to a list of information sets – perhaps Word™ documents, perhaps database-driven reports, perhaps tagged documents from a document management system – so the information is unified, right?

But it can be better than that. Think about the results you normally get back from a search engine: pages and pages of unrelated results. It would be nice if they were categorized according to people they were related to, or the cost of the goods discussed, or… well, or anything that you, as an end user, wanted them to be categorized by. Better still, they could merge together into something that looks like a report, or a “slice-and-dice” dashboard to help users to further analysis. You could find anything you wanted to in the form you need it.

It can’t really be that simple, can it?

Well, no. Most technologies that look simple on the surface are pretty sophisticated under the hood. We can start to see how complex enterprise search is by talking to early adopters, and they tell us that search projects are difficult to manage and the results often disappoint. A survey at a recent enterprise search conference showed that 59% of those with enterprise search solutions weren’t happy with their current implementations, and 25% of them are looking for a new search vendor.

It’s unlikely that these organizations will find a single solution that does everything they want it to. One large company said that they had deployed an industry-leading search appliance, but found that it was unable to fulfill the needs of many departments; as a result, different departments started to implement other solutions, and there’s now an expensive and complex enterprise-wide initiative to consolidate competing search implementations (i.e., migrate to a single solution) or federate them (i.e., provide a single user interface that aggregates results from all of them).

That said, some level of success must be possible, or people would have stopped implementing projects already. Members of the conference identified a lot of tasks you need to execute well to make enterprise search solutions work:

• Plan. Typical enterprise search projects allocate 40-70% of their effort to planning – and frequently fail when they don’t.
• Collect content. Although structured databases have dealt with extraction, transformation, and load (ETL) processes for years, managing complex unstructured content can be more difficult. Search solutions need to “crawl” document repositories, Web sites, and standard databases to acquire all of the information needed by end users.
• Manage index processing and retrieval. Enterprise search solutions create an “index” of all available information, so that when a user asks for information (e.g., types keywords into a search box) the engine knows where to find the documents that contain it. But how does it know that any particular document is more than a bag of words? Does it know where the document came from, whether there are security restrictions on it, whether it’s related to other information or whether it stands alone? Effective search solutions must provide that kind of information to the index as it’s being built, and must interpret it correctly upon retrieval.
• Create interfaces. Most people like to use Google’s interface and tend to distrust search solutions that differ from it; The screen is easy to understand and it provides them “good enough” results for many types of queries. System implementors must therefore identify ways in which their search solutions can provide “Google-easy” interfaces that deliver more precise and relevant results.
• Create navigation. The first user interface improvement people want is navigation – the ability to classify information and surf it using related categories instead of just getting a screen full of hits. This interface requirement, though, affects the collection and indexing requirements we’ve already discussed, and is affected by security considerations as well.
• Provide administration and analytics. In a complex environment with different collections of information, different sources of metadata, a wide range of queries, and so on, understanding the environment and providing the ability to optimize it becomes more important with everyone who gets added to the user community.
• Enforce security. Not everyone is allowed to see the same documents and reports, and even those who are allowed to see a given document may not be able to see all of the information in it. (For example, think of allowing certain people in Human Resources the ability to see an employee’s entire personnel record except for compensation-related fields.) This has traditionally (to the extent that we can talk of a tradition in this field) been a weakness – either an afterthought that requires substantial de­velopment and maintenance at each site, or a policy that limits users’ access to entire systems to protect subsets of the data.

In reviewing this list after the conference, I was struck by how many data warehousing and erp implementations have failed because they didn’t take these issues into account. They’re the same issues, applied to a different set of end user requirements.

And we know how a large number of those projects ended up: limping along to an underwhelming success, or even failing outright. Different architectures arose over the course of a decade or so to handle specific challenges, but no matter which one you looked at you saw repeated failures.

But if you looked closely, you could see incredible successes, too.

We have to go into these projects with our eyes wide open: Enterprise search may be a lot like data warehousing and erp. The key is finding ways to improve your odds of success.

The people who succeeded with data warehousing and erp were those who chose a project – a goal that needed to be hit, with an executive sponsor who had a problem and was willing to expend some money and political clout to get it. I’d suggest that the same is true here. If you don’t have a reason for enterprise search, if it’s just a cool technological idea, you should probably start looking for a different project to get behind.

When you do choose whom you’ll support most and why, then (and only then) consider what you’ll need to do in the areas above. Things will be different from what you’re used to.

For example, the content collection you’ll undertake with enterprise search solutions is a hybrid: neither the set-level extraction, transformation, and load (etl) needed for data warehousing, nor the transactional-level conversion needed for an erp conversion, nor even the content-driven document annotation used for document management systems, but a blend of all three. You’ll need to integrate quickly and tightly with hundreds of systems and thousands of messages (structured and otherwise) that flow through your organization. ETL isn’t enough, and neither is Web services.

(If you’re non-technical and don’t quite know what that meant, don’t worry – the point is just that it’s different.)

Collecting data is just the beginning. You’ll need to manage processes that catalog the information, enrich it with information from other systems, tag it with appropriate metadata and key words, provide security markers, and more. Although that can be tough (or at least tedious), doing it will allow you to convert some of your unstructured content into data that can be manipulated, reported on, even, possibly, sliced-and-diced.

If you’re like me, you’re thinking that things are pretty complicated for something that looks so simple. It’s starting to look as much like business process management (bpm) as it does enterprise search. And it’s true: it is complicated, politically and technically, like bpm or service-oriented architecture (soa). Despite that, your top priority is making enterprise search look incredibly simple. The harder it gets for users, the less likely they are to use the system that you just invested in. And the simpler you want to make your user experience, the more sophisticated your data, processes, and technologies need to be – because they have to handle every conceivable situation.

It’s tough, to be sure, but it’s possible. Our clients are getting their feet off of the ground by acknowledging that there’s a long way to go between typing in a few keywords and getting back the exact information you need to make a decision. They stay focused on what they need to do to make their implementation work, and we’re helping them go the distance.