In-memory databases are reaching a tipping point, says Microsoft’s Dave Campbell. Reduced prices for dynamic RAM, along with better compression and column storage, will be able to better help companies deal with their increasing crush of data.
His post on the subject, however, was seen as a “me-too” boast for Microsoft technology, given the recent attention garnered by SAP’s HANA and those of other players. Chris Hallenbeck, SAP’s senior director of HANA innovation, raved to me about HANA as “blazing fast” and opening up a whole new world of possibilities for business users.
With companies struggling to deal with their data – Hallenbeck put that growth at 40 percent a year – surely there are job opportunities here.
When vendor McObject began in 2001, co-founder and CEO Steve Graves knew of only two other vendors of in-memory databases, then known as “embedded” database systems. “Today, every small and large database vendor claims to have an in-memory database product,” he says.
Telecommunications, industrial control and financial services have been the forte of McObject’s eXtremeDB. Smartphones are among the uses for its lighter, open source product Perst. But McObject has also been working on dynamic programming for TV set-top boxes and MP3 players, a blowout-prevention system for deep-sea oil drilling, and navigation and mapping for combat jets.
Hype or Reality?
However, Mark Madsen, president of Third Nature, a consultancy in business intelligence and data warehousing, sees a lot of vendor hype.
“There’s a lot of interest in database technology,” he says, “but I see a lot of it being built into memory layers to traditional databases and then it being labeled as if it’s something spectacular.… And (vendors) seem to be setting it up as components of existing databases that you get to pay extra for, too.…
“Behind the scenes, you’ll see them built into more BI products. I hate to even call them in-memory databases because they’re really just reinventions of OLAP stuff from a long time ago, but Tableau, QlikView and MicroStrategy – all these guys have some form of in-memory technology.”
Companies in businesses like insurance, offering quotes on the phone in five minutes, could make use of the real-time or near-real-time results.
Where the Jobs Are
For now, jobs likely will be with vendors, Madsen says. “(Different) kinds of products, business intelligence, database, CEP (complex event processing), where they’re being used in the context of that product. I would expect more of that.”
Indeed, SAP is investing $155 million to encourage startups and entrepreneurs to focus on the development of real-time apps for HANA. Though offered as an appliance, there’s also a cloud version aimed at small- and medium-sized businesses, an area ripe for consultant work as well. And in Dice’s survey of 2012 hiring priorities, SAP professionals ranked No. 9.
The technology could affect these roles:
- Developers: Madsen says little will change with SQL-based systems, where each vendor has proprietary APIs in Java, C#, C/C++, or using multiple languages including Python or Ruby. Vendors also are doing much of the back-end integration – SAP has been working to make HANA talk to Hadoop, for instance. For companies such as Twitter and Google that use NoSQL systems for websites – MongoDB, Cassandra, CouchDB, Riak – Graves says for programmers and administrators, “these will necessitate an almost entirely new skillset (and mindset)” because they’re radically different from relational databases and from each other.
- Administrators: Many of the tuning tasks for increasing performance have been eliminated in in-memory systems — things like cache management and building indexes. That leads Hallenbeck to say fewer skills will be required. When asked the skills he would look for when hiring for an in-memory position at SAP, he pointed to those of a traditional DBA. Indeed, current job ads for in-memory positions focus on experience with Oracle Exalytics and TimesTen, Exasol, ParAccel, DB2, Sybase IQ, Teradata as well as related technologies such as BI and ETL tools. Graves says that just as the NoSQL systems will require some training for DBAs, so to will the more complex products from Oracle, SAP and Microsoft.
- Modelers: Hallenbeck says this work will change little, but that with a faster system, modelers will see fewer constraints and will be able to focus more on building the right model. They will have to know when about columnar tables versus row tables, one of the areas of training that SAP has in the works with HANA.
- Business users: Rather than limiting access to a few specialists, everyone in a company can be allowed access without affecting system performance, Hallenbeck says, providing huge opportunities to develop new services and lines of business. “Workers are going to be able to differentiate themselves substantially if they’re able to do analytics inside a system,” he says. “(So) if they have ideas about what they want to do, they can just go run that … It’s going to be an opportunity for people who are more junior to come up with ideas and really move the company forward.”