Harnessing the Web for Real-time Business Intelligence

It’s no secret. With the proliferation of web sites, blogs, news sites and feeds, there is a wealth of information available on the Web for Business Intelligence (BI) analysis, particularly in industries like retail, insurance, travel, and telecommunications. Let’s start with a general rule of thumb. When provided for commercial gain, information on the Web is easy to find. When mandated by the government for compliance, it might be easier to find deep sea scrolls.

Real-time, BI information is not gathered with a search engine. Business analysts generally know the sites where strategic and competitive data resides. The fun begins once the analyst gets to the sites and spends countless hours eyeballing, copying, and then pasting the data into a spreadsheet or database . A recent study by IDC suggested that information workers spend as much as half of their time searching and gathering information, at an annual cost of $26,700 per worker. A large organization could be spending as much as $5.7 million annually just copying and pasting information. Below are some ideas about how to reduce these tedious unproductive tasks, providing more time for actual analysis.

The low hanging fruit

Let’s take competitive pricing as an example. At first glance, retail sites would appear to be ‘easy pickins’. After all, they want you to buy on-line so pricing information is easy to find. Unfortunately, the pricing analyst can’t push a single button and get the whole price list neatly imported into a spreadsheet. Usually, the information is only available one piece at a time, either by category or product number. This imposes a significant manual burden to “unload” the entire price list as the analyst must make all the possible selections, then copy and paste one section at a time.

There are software products that automate this process. Called Web “harvesting,” “mining,” or “extraction” tools, some of them can intelligently simulate the manual selection process, automatically gathering all necessary information and transforming into the format of choice. Even better, the process can be scheduled to run during off-hours.


Those dreaded calculators

So that sounds easy enough, but what about the calculators business analysts run into occasionally? You’ve probably used them yourself for airfares, mortgage loan rates or insurance quotes. These calculators add another level of complexity as the final output is dependent on multiple variables: age, weight, smoker/non-smoker for life insurance and departing/arriving cities and dates for air fares. Because the input permutations are too huge to be done manually, the Web is often not considered a source for gathering these prices.

Not so fast! Sophisticated Web harvesting tools can overcome this hurdle too. Some are capable of discovering the forms calling for the variables and then simulating the necessary inputs from a table of variables stored in a local database or spreadsheet. After the data is retrieved or calculated, it can be automatically transformed into an actionable format. The Web harvesting tool turns an impossible task into something easily accomplished.


Putting your tax dollars to work

Organizations in highly regulated industries are required to regularly submit alls sorts of information to government agencies. These submissions are then made public on government Web sites. For example, in the pricing scenario, telecommunications companies are required to file pricing with the Federal Communications Commission (FCC). These submissions are made available to the public on the www.FCC.gov web site. For the pricing analyst, looking for competitor pricing it doesn’t get much easier than that, right? Not quite.

There are more than 100 of these lengthy documents containing information beyond product descriptions and pricing. Typically they are in PDF format, so they require downloading. To make matters worse, the submitters are not eager to make the pricing easy to find and understand, so the descriptions are cryptic and inconsistent. Finding the needles in these haystacks is so daunting, most telecom companies haven't even bothered - until now.

There are Web harvesting tools capable of discovering and automatically downloading these files into a large flat file of PDF formatted text. These tools can then parse the PDF text, searching for descriptions and data meeting prescribed criteria. Once found, all of the critical intelligence information can be extracted, reformatted into a database, spreadsheet, or even integrated into your BI solution.


The tradeoffs

All Web harvesting tools can gather information from Web sites that will make business analysts more productive. Some are very easy to use but sacrifice advanced features such as logical pinpointing, forms completion, scheduling, link analysis, downloading, and multiple file format support. Typically, these are the features providing the huge productivity gains and making the impossible tasks possible. Take a close look at your information gathering challenge - then make the ease of use versus capability decision.