Hadoop is all the rage in today’s big-data world, offering major companies the framework they need to crunch data. But with every major success comes the possibility of failure. And Hadoop is certainly not invincible.
Over the years, the enterprise has quickly changed its stance on many things. There was a time when the thought of having anything but a BlackBerry in the office was anathema, and to deliver Macs to employees was absolutely ridiculous. The corporate world—and the people that made the technology decisions—remained notoriously averse to change.
But things are different now. Both employees and their IT overlords welcome the iPhone and Macs with open arms. Despite security concerns, employees are given far more leeway with regard to their technology choices. The once-boring enterprise is now fickle.
So far, that fickleness hasn’t hurt Hadoop. In fact, Hadoop, the panacea in today’s big data world, is nearly universally beloved for its ability to crunch data into readily digestible parts.
But let’s not forget that Hadoop is competing in a space that has been known to have its fair share of wildly popular solutions that, for one reason or another, have lost ground. Its current popularity aside, there are several issues at play that could see Hadoop arrive at the same fate.
For instance, Hadoop is wildly popular with companies that are having an increasingly difficult time holding steady in the online world. Yahoo, AOL, and yes, even Facebook, are big companies now, but will they last? And if they go down, how will that effect Hadoop’s reputation—even if the framework had nothing to do with the market forces that drove those companies to collapse?
Facebook is perhaps the best test case of Hadoop’s abilities. Paired with the epic amounts of data running through the social network, Hadoop has been able to deliver results that supporters insist prove its worth.
But Facebook is a real threat to Hadoop. Social networks typically don’t last forever, and fickle users—there’s that word again—tend to go elsewhere after awhile. Facebook is starting to see some of that erosion in developed countries where its growth has stalled. If the worst happens and Facebook meets the same fate as its predecessors, that could be a huge blow to Hadoop’s standing.
By its very nature, Hadoop places its future in its users’ ability to stay afloat. While big companies use Hadoop to crunch growing amount of data, the framework has little chance of tumbling from its position as market leader. If those companies start failing, though, there’s an open door for more rival platforms to enter and gain market-share.
Hadoop is also no cakewalk for companies that want to invest in it. The technology requires a significant investment just to get up and running. Since it’s open source, the sheer number of ways it can be customized are nothing short of astounding.
But at what point does all of the complexity and customizability of Hadoop become troublesome for companies? Let’s not forget that implementing Hadoop means training employees on its use and dealing with the issues they’ll face as they work their way through the file system. It also means dedicating significant time and energy to its deployment, which can come at the expense of other projects.
For major companies such as Facebook or Yahoo, deploying Hadoop and investing in it isn’t necessarily a problem. But it’s different for small companies. As useful as Hadoop might be, the technology presents several costly barriers to entry that not only hold back its future chances of success, but also diminish the overall ability for companies to handle their biggest data.
Perhaps that’s why so many Hadoop competitors have cropped up. The framework, in fact, is flanked by solutions from companies such as Nokia Research Center’s Disco. And let’s not forget that massively parallel processing is still capable of handling Big Data needs.
Hadoop’s current saving grace is that it’s being integrated into many of the solutions offered by would-be competitors. But whether that will last over the long term remains to be seen.
Whether the legion of Hadoop fans want to admit it or not, Big Data is big business. Right now, Hadoop and its file system—which, let’s be honest, can be a pain to use at times—are necessary components for companies that want to take advantage of the Big Data craze. Over time, those companies could ditch Hadoop for their own, fully owned, alternative. It’s not that they can’t stand Hadoop; it’s that for-profit companies that operate in the enterprise love to have control—and Hadoop doesn’t allow enough control for them to be happy.
So, where might all of this leave Hadoop? For now, in the same place. The framework is still the top gun when it comes to crunching massive amounts of data, and most would agree that it does a lot of things really well. Still, market factors are a big concern for the technology; and unfortunately for Hadoop, it’s not operating in a vacuum. For now, stakeholders have a vested interest in ensuring that it stays atop the space. But as the market changes, big users feel increased pressure, and competitors become less willing to use the technology, Hadoop might find itself in the unenviable place of fighting for attention against some up-and-coming rivals.
As noted, the enterprise is a notoriously difficult space to succeed in over the long-term. Big data is a big problem and a big opportunity for companies. And Hadoop, for now, fills the gap. Yet nothing is set in stone, and given its challenges, it’s hard to say for sure that Hadoop can hang in there for the long haul.
In just a few short years, we might be having a much different discussion on big data. And Hadoop, with all of its bells and whistles, might not play such an integral role in that talk.
Image: Jose Luis Mesa/Shutterstock.com