Demand for Big Data talent certainly seems strong, but what's driving it? Some argue there's a talent shortage, but it seems that the hiring process might play a role as well. Employers may not be asking the right questions in the interview that will help them hire the right candidates. There's no doubt that the past year has been a wild one when it comes to job opportunities in Big Data. There is a dramatic shortage of Big Data technology professionals, and this shortage is expected to get worse with demand exceeding 200,000 technology professionals by 2018, according to reports by Gartner and the McKinsey Global Institute. In my own personal experience, I've had 10 to 20 headhunters and technical recruiters contacting me each week since February. So, I can attest that there is a great deal of demand – or so it seems. As an experienced technical professional and very senior manager in an area that has become known as Big Data, I have been fortunate and perhaps unfortunate to be on both sides of the hiring table. Fortunate, because I too have sought to hire technology professionals for positions in Big Data. Either, I've been able to find candidates with the right skills or find technology professionals with the right foundational background and passion that I can cultivate. So far, I've been quite lucky in that all my new hires have performed extremely well and integrated well with my current staff.

Poor Recruiting Practices

Unfortunately, I've been approached by countless headhunters and technical recruiters looking to recruit me for companies that they may or may not represent. The recruiter who doesn't represent the opportunity or the company is wasting my time. In one case, a ‘recruiter’ tried to have a cold call with my own company’s HR staff and as part of that cold call sent them one of my own resumes from 10 years prior as proof that they could recruit the right staff — tsk, tsk. Of the legitimate recruiters, the vast majority have asked me to apply for Big Data infrastructure, engineering or data analyst positions. Most of these positions require 5 to 8 years of experience. And while I certainly could do any of these positions, I'm probably not the best match for positions so junior. On my resume and LinkedIn profile, it actually shows Big Data technical experience dating back to the 1990’s with my current position as a director.  Frankly, I chalk it up not so much to inexperience as laziness. This type of recruiter relies on search engines instead of thinking. My company’s own HR recruiters have been guilty of this same practice.

The Right Questions for the Job

Once I've gotten past the HR recruiters and start the interview process, I find that most of my interviewers are senior managers, directors, and VPs.  This is what I'd expect at this level. What I didn't expect was that many of the interviewers either had no idea about the technical or business purpose of Big Data or were briefed to ask questions appropriate for a mid-level Big Data-Hadoop engineer. In one case, I was interviewing for a Chief Data Scientist position and not one interviewer asked any questions about briefing non-technical business decision makers on what a technical analysis meant for the business. In the absence of this line of questioning, I proactively offered how I handled this type of briefing based on an actual briefing I had conducted with one of my current customers.

Anticipate What the Interviewer Needs to Know

I've designed, architected, and built infrastructures and teams to create Big Data and analytical systems and products. I've provided analytics that decision makers have used to make informed business decisions. Not one of my interviewers asked about this experience, even though I was applying for senior manager or director level positions that involved this type of activity and responsibility. Again, I wove accounts of how I had accomplished this in the past into my answers to questions they were asking. This year, I've been the finalist for six senior positions in Big Data. This usually meant that I was one of the final three. The interview process usually took about 2 months and involved 8 to 10 interviewers. I was offered the job for one of these positions but, as this was a start-up, could not come to an agreement on compensation. So were these folks asking the right questions? No. But I answered based on the responsibilities listed in each job description. Not only did I answer the interviewers' questions, but I also anticipated what they really needed to know. I provided answers to questions that directly related to the job description and my own experience in Big Data. Was this the right thing to do? I don't know, but putting on my ‘hiring manager’ hat, this is information that I would think is necessary to make an appropriate decision.

Big Data Gold Rush

Are there many new opportunities in Big Data? Absolutely! For those of us with a Big Data background, I equate this to the California Gold Rush. As an employer, I may not always find someone with the right experience. But, since I know the Big Data area very well and what it takes to deliver, I look for technical people with the right foundational background and who have a passion that can be directed to solving problems and seeing patterns in information. These types of individuals make the best people for Big Data. On the other side, many companies either have Big Data programs or are trying to develop Big Data-related products. I've found that very few recruiters know what they are looking for, mostly because the hiring managers do not truly understand the technical and business requirements. Will this get better? I really don't know. I sure hope so. Finally, I do hope that certain company recruiters – especially those from the San Francisco Bay area's peninsula — when doing a pre-screen for a senior manager or director-level position, stop asking ridiculous questions that only a practicing UNIX/Linux Systems administrator would know. The impression it gives of their company isn't a flattering one, especially for senior-level positions.