Is Hadoop an Overhyped Technology?


For years, Apache Hadoop appeared ubiquitous within the data-analytics community. Every week, it seemed, more companies voiced their reliance on the open-source framework, ideal for running data applications on large hardware clusters; software vendors offered up proprietary distributions, with claims of improved ease-of-use and faster processing speeds.

But has the actual implementation matched the hype? Pivotal, the Big Data-centric spinoff of EMC and VMware, announced back in February that it would stop producing its proprietary Hadoop distribution, perhaps (Venturebeat surmised) in response to a market crowded with similar platforms. Yet despite all the distributions available, a new Gartner report suggests only a minority of companies are actively deploying or experimenting with Hadoop.

In a recent Gartner survey, only 26 percent of respondents reported either deploying, piloting, or experimenting with Hadoop; another 18 percent said they plan on investing in the technology at some point within the next two years. Many organizations have downgraded Hadoop’s importance in relation to other software initiatives, and others have claimed the associated costs are too high.

“Early Hadoop projects typically involve a small number of users and this no doubt keeps user populations down at this stage of the market,” Nick Heudecker, research director at Gartner, wrote in a brief research note. “Moreover, the Hadoop stack remains unsuitable for simultaneous use by multiple users, also keeping numbers down. Another factor, and possibly the best explanation for the small number of Hadoop users, is the skills shortage.”

Whether or not Hadoop eventually lives up to its longtime hype, data analytics has become an increasingly vital element of organizational strategy. Tech pros with a background in the various Big Data technologies can easily find themselves in high demand. But which analytics skills pay the best? According to Dice, the top seven, by annual salary, include:

Cassandra: $128,646
MapReduce: $127,315
Cloudera: $126,816
HBase: $126,369
Pig: $124,563
Flume: $123,186
Hadoop: $121,313

Whatever you say about Hadoop, it can still prove lucrative.

3 Responses to “Is Hadoop an Overhyped Technology?”

  1. Fred Bosick

    *There is no such thing as a skills shortage for a software package whose initial release was in 2011.*

    Just because some proselytizer or clueless bandwagon manager thinks it’s the greatest thing since ever wants some “Hadoopalizer” to drop into his IT department, doesn’t mean that it the fault of job applicants who didn’t immediately download the package and became an instant expert. It needs time to grow and become visible to many IT aspirants as a concept worth looking into.

    IBM didn’t hire COBOL programmers in the 50s; they trained them. Want Hadoop experts? Train them! Or find a package better suited to your business needs, and then train longtime IT workers, or just graduated CS students.

    SourceForge has a *lot* of packages to do various things. Who knows what the next big thing is? Don’t expect instant experts in some software that journalists say is good for big data or big whatever.

  2. Geo Wade

    While in graduate school, I gave a presentation in front of the class in my Database: Security and Auditing course on ‘Big Data’. Specifically on a NoSql database technology, and its stack. I had chosen the Hypertable NoSql db. During the research for the presentation I had thought to myself that gaining experience using the Hypertable, Hadoop, and Map Reduce; could align a data architect with a possible position at Google, considering that the Hypertable is modeled after Google’s BigTable.

    In the discussion of Hadoop, there is more to be considered than just Hadoop itself. Alot of knowledge is to be gained with the overall stack when considering employment and possible of avenues of development.

    Just food for thought.