Coping with Too Much Data: How Boeing, Nike and Others Did It
Businesses wring their hands over having too little data. But what happens when you have too much of the same data? Figuring out conflicting reports, deciding between different metrics, and removing duplicate entries can prove an enormous drain of time and resources—especially for some of the world’s largest companies, which have implemented too many data warehouses, or data marts, that tell different stories about the same business processes or events.
Every executive wants workers to run reports that present accurate and consistent information—no matter what the data’s origin. At this month’s Teradata User Conference in DC, I heard from a number of IT architects on how they handled the situation and got their data more “truthy,” as Colbert might say.
I saw three different approaches towards this goal at the conference:
First, reduce your numerous data warehouses to one instance. That is what Boeing and Land Rover are doing.
“We had disparate data marts and needed expert knowledge to interpret our data,” said Simon Leese, data architect with Jaguar Land Rover. The company saw lots of inconsistencies in its data; Leese told stories of users who would spend time debating the different numbers in seemingly endless meetings: “We were missing business opportunities. The current environment wasn’t able to support any growth and couldn’t adapt easily to business changes or to future-proof our business.”
He added: “It is important that information is in one place and that it is usable.” Land Rover worked with Teradata’s manufacturing modeling team to simplify its data mart to a single integrated view of their business, a project that took about a year to complete.
Boeing went through the same process, starting with 12 data warehouses and 50 cost management systems, some of which had tens of thousands of business rules. “The problem was that our IT department was doing a great job of giving their users what they needed, but they weren’t talking to each other,” says Bill Curley from Boeing’s finance department. This lack of integration was causing all sorts of inconsistencies with their reports.
It took Boeing several years to consolidate all of its financial reporting into a single consistent place. Those working on the issue took a top-down approach, asking what information was needed for data owners to actually do their jobs, and implementing a standard data dictionary that had the minimum of elements needed. They also separated elements that were operational from the actual accounting data that was needed for their reports. “We didn’t need to carry the operational information through our accounting system anymore,” Curley said.
Ironically, the end result was a very simple system. “Our data dictionary now is very user-friendly and could be easily updated and be a living document,” he added. It was implemented using Microsoft Access with a Web front end.
Second, use more of a unified data architecture as a motivation to improve data quality. This is what Nike has been working on for the past several years. At the conference, Nike data architect James Lee showed how he eliminated duplicate data, filled in missing required data elements in his tables, and made over a series of reports that took too long to process into results.
“In our road towards having one version of the truth, we wanted to have plenty of flexibility so business owners could create their own reports without any IT intervention,” he said. That involved a goal of user self-sufficiency about their data: “We were telling them to continue to make their own bread, rather than stocking their pantries with the right ingredients that they could assemble on their own.”
One of the most commonly accessed tables at Nike had more than 100 columns. It was tremendously inefficient from an input/output standpoint, on top of wasting processing resources. Nike simplified these ultra-wide tables and cut down their data models into fewer elements. The process also improved their data quality, as end users took more responsibility for the missing-but-required data elements and became more proactive at tracking them down.
Lee also realized that there is a right and wrong way to go about this unified data process: “The wrong way is to put 20 people in a conference room and try to have them figure it out in a meeting. Instead, I got them to give me their most accessed reports and found the optimal way to collect this information and present it to them.”
Third, consolidate your demand chain management. While supply chain management gets a lot of attention, companies should also consider how to best track the demand for their products.
This can be especially vexing when it comes to handling retailers who have both online and brick and mortar presences, such as Macy’s, which operates two online business units alongside three additional units running 800 physical stores around the country. Given its size, you can understand the problem: what if a customer buys an item online but wants to return it to a physical store? Or wants an item that they see online but isn’t in stock in their nearest store?
“We wanted our customers to buy anywhere and be able to fulfill the order from anywhere,” said Wade Latham, the Director of Business Process at Macy’s. The problem was that their original processes were mostly manual or used Excel spreadsheets to track demands. “We couldn’t recognize seasonal or climate differences among our stores, and couldn’t really accurately forecast inventory levels. We also wanted to collaborate and share information both internally with our merchants and externally with our vendors for better planning, so they would have the product to ship us when we need it.”
Macy’s switched to Aprimo’s Demand Chain Management software, and used several of its retail-specific modules for intelligence stock item introduction monitoring and tracking clusters of item profiles. “We focused on the opportunities surrounding replenishment of our stock, because they have higher profit margins,” Latham said. “Now we account for seasonality and can rank our stock items by location and know exactly what inventory we have on hand.” About 40 percent of Macy’s stock has been entered into the new system, which took about 18 months to build from start to finish. Latham says Macy’s is seeing a seven percent sales increase and more frequent inventory turns as a result—not to mention an increasingly satisfied customer base.
All of these methods involve what Teradata Labs’ president Scott Gnau calls finding the “golden path” to your data. He told me that many IT data architects are looking at making it easier and quicker for end users to get reports and reducing inconsistencies in them. Eliminating duplicate data warehouses or creating consistent data architecture are all good ways to progress down this road.