Big Data Talent Shortage? Is Recruiting to Blame?

Demand for Big Data talent certainly seems strong, but what’s driving it? Some argue there’s a talent shortage, but it seems that the hiring process might play a role as well. Employers may not be asking the right questions in the interview that will help them hire the right candidates.

There’s no doubt that the past year has been a wild one when it comes to job opportunities in Big Data. There is a dramatic shortage of Big Data technology professionals, and this shortage is expected to get worse with demand exceeding 200,000 technology professionals by 2018, according to reports by Gartner and the McKinsey Global Institute.

In my own personal experience, I’ve had 10 to 20 headhunters and technical recruiters contacting me each week since February. So, I can attest that there is a great deal of demand – or so it seems.

As an experienced technical professional and very senior manager in an area that has become known as Big Data, I have been fortunate and perhaps unfortunate to be on both sides of the hiring table.

Fortunate, because I too have sought to hire technology professionals for positions in Big Data. Either, I’ve been able to find candidates with the right skills or find technology professionals with the right foundational background and passion that I can cultivate. So far, I’ve been quite lucky in that all my new hires have performed extremely well and integrated well with my current staff.

Poor Recruiting Practices

Unfortunately, I’ve been approached by countless headhunters and technical recruiters looking to recruit me for companies that they may or may not represent. The recruiter who doesn’t represent the opportunity or the company is wasting my time. In one case, a ‘recruiter’ tried to have a cold call with my own company’s HR staff and as part of that cold call sent them one of my own resumes from 10 years prior as proof that they could recruit the right staff — tsk, tsk.

Of the legitimate recruiters, the vast majority have asked me to apply for Big Data infrastructure, engineering or data analyst positions. Most of these positions require 5 to 8 years of experience. And while I certainly could do any of these positions, I’m probably not the best match for positions so junior. On my resume and LinkedIn profile, it actually shows Big Data technical experience dating back to the 1990’s with my current position as a director.  Frankly, I chalk it up not so much to inexperience as laziness. This type of recruiter relies on search engines instead of thinking. My company’s own HR recruiters have been guilty of this same practice.

The Right Questions for the Job

Once I’ve gotten past the HR recruiters and start the interview process, I find that most of my interviewers are senior managers, directors, and VPs.  This is what I’d expect at this level. What I didn’t expect was that many of the interviewers either had no idea about the technical or business purpose of Big Data or were briefed to ask questions appropriate for a mid-level Big Data-Hadoop engineer.

In one case, I was interviewing for a Chief Data Scientist position and not one interviewer asked any questions about briefing non-technical business decision makers on what a technical analysis meant for the business. In the absence of this line of questioning, I proactively offered how I handled this type of briefing based on an actual briefing I had conducted with one of my current customers.

Anticipate What the Interviewer Needs to Know

I’ve designed, architected, and built infrastructures and teams to create Big Data and analytical systems and products. I’ve provided analytics that decision makers have used to make informed business decisions. Not one of my interviewers asked about this experience, even though I was applying for senior manager or director level positions that involved this type of activity and responsibility. Again, I wove accounts of how I had accomplished this in the past into my answers to questions they were asking.

This year, I’ve been the finalist for six senior positions in Big Data. This usually meant that I was one of the final three. The interview process usually took about 2 months and involved 8 to 10 interviewers. I was offered the job for one of these positions but, as this was a start-up, could not come to an agreement on compensation.

So were these folks asking the right questions? No. But I answered based on the responsibilities listed in each job description. Not only did I answer the interviewers’ questions, but I also anticipated what they really needed to know. I provided answers to questions that directly related to the job description and my own experience in Big Data.

Was this the right thing to do? I don’t know, but putting on my ‘hiring manager’ hat, this is information that I would think is necessary to make an appropriate decision.

Big Data Gold Rush

Are there many new opportunities in Big Data? Absolutely! For those of us with a Big Data background, I equate this to the California Gold Rush. As an employer, I may not always find someone with the right experience. But, since I know the Big Data area very well and what it takes to deliver, I look for technical people with the right foundational background and who have a passion that can be directed to solving problems and seeing patterns in information. These types of individuals make the best people for Big Data.

On the other side, many companies either have Big Data programs or are trying to develop Big Data-related products. I’ve found that very few recruiters know what they are looking for, mostly because the hiring managers do not truly understand the technical and business requirements. Will this get better? I really don’t know. I sure hope so.

Finally, I do hope that certain company recruiters – especially those from the San Francisco Bay area’s peninsula — when doing a pre-screen for a senior manager or director-level position, stop asking ridiculous questions that only a practicing UNIX/Linux Systems administrator would know. The impression it gives of their company isn’t a flattering one, especially for senior-level positions.

8 Responses to “Big Data Talent Shortage? Is Recruiting to Blame?”

  1. Fred Bosick

    “Big Data” is a nebulous term. No one can say what skills make it up. So anyone recruiting big data professionals has no clue. Better to look for people based on what the organization is trying to do with all this data. Massive hardware? Enterprise. Databases? Oracle or SQL. Etc.

    The problem isn’t finding the people, it’s hiring. A brand new discipline or specialization doesn’t have ready made people about to walk into the door after a phone call or interview. The organization might, gasp, have to train or let the new guy learn on the job! Horrors!

    There’s no shortage of people willing to learn. However, there *is* a shortage of people who will incur all the opportunity cost to learn an unknowable list of skills hoping that they get picked up only to be laid off as soon as the company thinks it’s become a commodity.skillset.

    • The term “Big Data” is a big buzzword used by many in media and it has been adopted by the tech community. “Big Data” is NOT new. We used to call this High Performance Technical Computing (HPTC) with advanced analytics. Long have my colleagues and I used data bases, search engines, and parallel or distributed computing to sift through lots and lots of data looking for patterns or for conducting experiments looking for answers to questions. There is nothing new about this. I disagree that it is nebulous.

      “Big Data” involves the three V’s with respect to data:
      1) Volume – large volumes of data
      2) Velocity – the data comes in very fast or it must be processes very fast
      3) Variety – the data comes in all varieties and covers structured, unstructured, and semi-structured data

      IMHO, the field of “Big Data” is evolving – as it should be.

      Do I find candidates with the right skills to work on Big Data projects? Sometimes yes but usually not. I do look for folks with foundational skills, a passion for working with data and information, and an eagerness to learn and work as a member of a team.

      Is there a shortage of people willing to learn? It’s hard to say but candidates who are not already looking for and implementing new ways of improving their own skills, I have to say that I have very little interest in them as an employer as it shows that they are not taking care of their own skills – their own knowledge set. And as an employer, I do encourage my staff in outside learning opportunities either through Master’s programs, seminars, or other methods of training.

      Sure there are employers who expect folks to come in already trained. If they are hiring for “Big Data” programs and projects, they are also very naive. Many of these same employers do not even understand what they plan to use “Big Data” for and how it can be an enormous competitive advantage.

      So is there indeed a shortage of talent or is it the hiring process? Personally I think it is both. If you want to work in this area of “Big Data” start by understanding what it is (in general) and what skills are necessary because really it is your skills, your knowledge, and your passion for technology that you can control.

      As for recruiters and head hunters, I really do not know what to say. I too find them puzzling and frustrating – and that’s dealing with them on both sides of the hiring fence. But then again, every once in a “Blue Moon” you actually find a recruiter who knows what he is talking about…

  2. To @ME, was there a question in your comment? I think I already said that “Big data” is NOT new.

    For those of us already in this area, its nice to be in such demand. For those of us looking for new hires and already know what we are looking for, there are opportunities on both side. You just have to want to take that first step.

      • Job search enignes are the obvious first place to start for any jobseeker, however, the competition is fierce and those old ways just won’t work as well for you anymore. It sounds like you have great skills and will be a wonderful hire for a lucky company, but in order for them to find you, you had better stand out in the crowd. Consider things like personal branding and unique tools like VisualCV but most importantly, make sure you get out and network. Find an IT networking group in your local area and make the time to attend meetings and functions. Networking is the key to the hidden job market that most people never get right. As a last resort, find a company that may be looking for interns and get your foot in the door. It’s that time of year. Once you have the opportunity to prove your value, the offer may just be right around the corner! Good luck!

  3. Excellent article.
    I generally think that the Managers/PMs/PMOs/Sr. Recruiters that are reaching out in this area are extremely junior with what Big Data (mostly Hadoop on cheap boxes) actually is. Upper management reads a Forbes article on Big Data and then the recruiters start jumping on the phones.