Inside many large organizations, there exists a cadre of people tasked with transforming requests for information by business users into a query that an application can understand. The organizations would have a hard time functioning without these people, who are generally known as business analysts. In fact, many of these analysts are widely regarded as some of the most highly valued employees within their organizations. But with the rise of Big Data, that’s all about to change. As organizations begin to develop their first Big Data applications, many of them are discovering there isn’t going to be a need for a business analyst conversant in SQL to generate a report. Instead of having to store data in a data warehouse where it can be queried using SQL, they can envision a world where data-mining algorithms inside the Big Data application will identify the right patterns or trends. Take the example of Caserta Concepts, which creates applications that leverage the Apache Hadoop framework to process raw data—skipping the data-warehousing stage altogether. “The thing you have to remember is that, before all that data in the data warehouse became structured, it was already in an unstructured format,” said CEO Joe Caserta. “There’s no requirement to structure data if you can work directly against the raw data.” The end result, he added, isn’t just a reduced reliance on data warehouses; it’s an elimination of the need to have a business analyst translate requests between the business user and the raw data. In addition, there’s no need to waste time and effort building the schema definitions required to generate reports in an SQL database environment. Ted Dunning, chief application architect for MapR Technologies, which provides a Hadoop distribution optimized for real-time analytics applications, suggests that, instead of building reports based on (for example) 12 points of data, the algorithms under development will essentially observe changes to the same raw data over and over again in order to identify patterns and anomalies. It will take a few years for all this to play out, largely because of the current shortage of data scientists. But Dunning believes much of that shortage exists not because there aren’t enough people with doctorates, but rather because we don’t have enough people trained to think logically: “We need more people that know how to actually listen to the data.” Jim Kaskade is CEO of Infochimps, a cloud service that makes a distribution of Hadoop available to developers as a service. He concurs with Dunning’s opinion: “I’ve already seen people with little to no formal training accomplish a lot more than someone trained in legacy business intelligence and data warehousing applications, simply because they didn’t let legacy technologies limit their thinking.” But that lack of experience also leaves many organizations concerned that the Big Data projects they launch will fail to live up to their potential. In fact, a recent survey of 300 IT people conducted by Infochimps found that 55 percent of Big Data projects are never completed. To insure success, Kaskade says IT organizations need to focus on a specific business problem that the app can solve: “Don’t try to boil the ocean.” Steven Hillion, chief product officer at Alpine Data Labs, a provider of analytics applications, sees the same issue emerging. “There are clearly not enough data analysts that are really data savvy or have the right tools,” he said. “A lot of these Hadoop projects that people are working on today don’t even really involve big amounts of data.” But because Hadoop is the hot new thing, Hillion thinks a lot of IT people are trying to add Hadoop skills to their resume without actually addressing a real business problem. Of the 800 executives recently surveyed by Connotate, a provider of Web data monitoring tools, some 80 percent felt it was too early—if not outright impossible—to measure Return on Investment (ROI) from Big Data tools. Nonetheless, a vast majority of the surveyed enterprises were moving ahead with their respective projects, targeting a variety of objectives. Nearly 50 percent were using Big Data for competitive monitoring; over 40 percent cited brand monitoring, while 38 percent indicated the goal of obtaining pricing and product information. “What’s really interesting is the amount of what we call Black Data that organizations have not used for years that is now being repurposed in Big Data applications,” said Connotate CEO Keith Cooper. “Of course, it shouldn’t really be about Big Data versus small data but rather identifying important data.”

The Business Analyst Transformed

While everyone agrees the role of the business analyst is transforming, no one is quite sure what the future holds. Rich Rodts, manager of analytics academic programs at IBM, believes that rise of Big Data analytics may not lead to the demise of business analysts as we know them today, but rather data analysts reinventing themselves. But he also conceded that, as business users become more IT savvy, they will want to directly interrogate raw data themselves—which means business analysts will have to find another way to add value to the organization. “I don’t think business analysts are going away as much as evolving,” he said. “But I’m not sure what we may wind up calling them.” As business users become more accustomed to working with analytics applications, many of them will want to ask questions—and based on the answers, immediately ask other questions without having to rely on an analyst to translate their request. Rodts added that, given evolutions in the field of cognitive computing—most notably in the form of the IBM Watson platform—machines could even provide answers to a whole range of anticipated questions before they’re asked. Then there’s SAP, which contends that the ability to speedily interrogate massive amounts of data is one of the primary reasons that applications (and the underlying databases that support them) should run in-memory. SAP has a vested interest in that technology, of course, given its massive investments in its High Performance Analytics Appliance (HANA) in-memory computing platform. The company argues that, by folding the reporting process back into the transaction processing system (rather than having it take place in a traditional data warehouse), analytics will become orders of magnitude faster than current business-intelligence applications based primarily on disk-based systems. Of course, whether business analysts evolve into data scientists or some other form of “data whisper” remains to be seen. What is for certain is the job of the business analyst, at least as it’s currently described, has a limited future.   Image: Elena Elisseeva/Shutterstock.com