Architecting for Internet Retail Success Requires a BSM Cornerstone

In1997 Lee Lorenzen, founder and chairman of the board for SHOP•COM, found his mailbox bulging with more than two dozen catalogs, all delivered in a single day. Being both an entrepreneur and a technologist, it occurred to him that there must certainly be a more efficient way for merchants to place their offers in front of customers, and he was barely back to his front door before he’d figured out how.

Instead of “pushing” offers to customers and hoping the products and timing matched their needs, why not invent a way for customers to “pull” offers for the exact products they wanted precisely when they wanted them? For centuries, buyers and sellers have come together in marketplaces of one type or another, because marketplaces are more convenient for customers and more efficient for merchants. Perhaps this emerging technology called the Internet could make it possible to build a virtual marketplace? From that question a company was born.

Now, backed by a decade of technology development, the original vision has evolved into an extremely robust multi-merchant marketplace where customers can shop from thousands of brands and hundreds of stores and brands using OneCart™ and a single account. As brilliant as it is simple, it’s called SHOP•COM.

Growing Pains

About two years ago, SHOP•COM was in the process of dramatically increasing its product base by adding hundreds of top-brand merchants providing millions of new products for consumers to purchase. The idea was to create an environment where both merchants and customers want to be. After all, customers go where they have the most choice and merchants go where the customers are. So, SHOP•COM started with merchants. Lots of merchants... all in one place. Big merchants. Small merchants. Merchants with household brands. Merchants with niche products.

However, the addition of these merchants also meant that traffic to SHOP•COM was going to increase significantly as consumers found more to purchase. The challenge for SHOP•COM’s IT team was to scale the system to handle five million additional products and scale the web site to handle more than a doubling in traffic – and do all this on a shoe-string budget with minimal resources. Essentially, do more with less. IT operations was also tasked with finding a way to dramatically reduce the amount of downtime, increase site performance, significantly reduce problem resolution times and maximize returns.

Lucky for SHOP•COM, the company employs some very smart people. An engineering company at its core, SHOP•COM built more than 90% of its infrastructure in-house, including its customer-facing application and search engine. Between 2004 and 2005, the IT infrastructure had doubled in size, though without the luxury of doubling headcount. To support the added content and increased traffic, IT operations scaled a number of components, purchased several fully-loaded HP Blade enclosures, and integrated several new SAN systems. The IT/engineering team also employed database clustering techniques, and the whole system is load balanced and highly redundant, including a co-located facility. IT operations also deployed new software for the cache server, load balanced search and database servers. While solid, there was no way to tie in all the information from SHOP•COM’s home-grown management tools to find the root cause of performance issues, not just availability issues. Additionally, the IT management tools they did possess, while quite sophisticated, did not have the capability to provide early warning of developing problems.

“IT management products have evolved from silo-specific availability monitoring tools, to tools that track the end-to-end IT infrastructure,” said Julie Craig, senior analyst, Enterprise Management Associates. “While availability monitoring is necessary, it’s not enough. Next generation end-to-end tools must provide real-time visibility into the cause of hard-to-diagnose performance problems such as brownouts or intermittent anomalies—the main affliction of today’s complex, multi-tiered computing environments.”

To track the end-to-end performance of the IT infrastructure, then correlate and analyze that data to help identify and resolve developing performance problems, a business services management (BSM) solution from ProactiveNet was selected. Using ProactiveNet, staff can link custom business metrics (such as the number of online transactions per minute) to the performance of the IT infrastructure in real-time.

“It’s incredibly helpful when your management tool can spot a problem before it happens,” said Vince Hunt, executive vice president of engineering, at SHOP•COM. “ProactiveNet’s management console will indicate, ‘You’ve got a problem brewing in Server Farm A, and among those 50 servers, number 38 is the likeliest culprit.’ The product tracks performance in real-time on a site that serves over a million sessions a day. We’ve eliminated finger-pointing meetings, and most importantly we’ve eliminated downtime. During the holidays, that can mean hundreds of thousands of dollars in revenue savings per hour.”

A Winning Solution

During the 2005 holiday shopping season, the degree of success in terms of measurable, percentage-based ROI, cost-savings and challenges addressed exceeded all expectations. Sales volume increased 190%, traffic increased 260%, and the number of products increased 300%. Yet it was the smoothest holiday shopping period in SHOP•COM’s 8-year history. With the help of ProactiveNet, downtime was 100% eliminated, and early warning of performance problems combined with ProactiveNet’s correlation and analytics engine helped staff resolve developing issues before they impacted site performance. As Mr. Hunt indicated above, holiday revenue savings were estimated in the hundreds of thousands of dollars per hour.

The engineering/IT staff at SHOP•COM believe that their in-house developed infrastructure coupled with ProactiveNet’s BSM solution enables them to provide a site that rivals the world’s largest e-commerce sites—but with only 10% of the IT headcount of the big guys. In fact, the team was so proud of the innovative way they built their IT infrastructure—after all, it enabled the customer facing site to perform flawlessly during the holidays while increasing operations efficiencies—that they decided to enter two industry contests to tell their story: Retail Systems Magazine’s 2006 Awards for the “Best in IT/Business Alignment” category; and Computerworld Magazine’s 2006 Best Practices in Infrastructure Management Awards in the “Managing to Improve TCO/ROI” category. As it turned out, judges for both contests agreed with what SHOP•COM had felt all along, which is, they had built an exceptional infrastructure to support an exceptional site. The judges, consisting of industry luminaries from some of the top IT industry research firms, trade magazine editors, and peers from Fortune 1000 companies, recognized SHOP•COM as the winner of Computerworld’s Best Practices in Infrastructure Management, and as a finalist in Retail Systems’ Best in IT/Business Alignment.

“With hundreds of millions of transactions a year and growing, our biggest infrastructure challenge is to manage that growth without breaking the bank,” said Geoff Caras, vice president, system infrastructure, at SHOP•COM. “ProactiveNet proved to be an essential part of that strategy. The product automates many of the tedious tasks associated with end-to-end infrastructure performance monitoring, gives us a heads-up when there’s a problem brewing, and provides us with actionable information that helps us identify and solve issues quickly. When you couple automation and scalability, you can’t lose.”