Nailing the Next-Generation NOC

by Dice Staff Aug 19, 2009 5 min read

The tools in today's network operations center (NOC) may be state of the art, but oftentimes the training, mentoring and overall approach need an update.

By Mathew Schwartz

Today, an average 80 percent of a company's IT budget goes to operations. But according to Forrester Research Analyst Glenn O'Donnell, "a frightening proportion (of that) is wasted by inefficiency." The chief culprit? Too many IT functions operating in isolation, he says, a problem often compounded by business leaders' response: punitive outsourcing.

While the concept of networking has evolved to encompass everything from Ethernet to e-mail and SharePoint to cloud computing, and while IT has dutifully begun monitoring such activities via the network operations center (NOC), many organizations haven't rethought the NOC concept itself. "Twenty years ago, when I started doing networking, if you'd told me that on the same wire that I'm running my data across, I'm going to carry video, voice, electric power, I would have said you've got to be frickin' crazy," says Jimmy Ray Purser, a network engineer for Cisco Systems. "But sure enough, that's what we're looking at. Our jobs have been getting more concentrated, and the NOC should reflect that."

<pOperations: The State of the Art

Techncally, modern operations centers and monitoring tools - including APIs for "rolling your own" - make it easier to predict when something will fail, and why, says Purser. But while the tools have matured, often the business approach has not. When it comes to the NOC's Tier 1, too many managers think cheap talent, no training. They watch only for when red lights go green, then escalate the problem. It's "just a step up from 'would you like fries with that?,'" says Purser, which exacts a heavy toll on employee morale. "They're failing, not liking it, and quitting IT."

"Cheap talent" thinking also misses the point. "If I can take any NOC operator and replace them with a shell script, then I haven't trained them," says Purser. Creating an efficient NOC, and high levels of morale, requires education.

Build a Better NOC

How can organizations make their NOC more efficient, and better groom entry-level personnel for more advanced positions? Start with these five steps:

Sharpen Skills: Train Tier 1 personnel. If your company can't afford outside help, Purser recommends having your Tier 2 (specialists) and Tier 3 personnel ("the geek's geeks," in many cases outside consultants) design a training course. Just one week's training, he says, will increase uptime and reduce the number of problems that must escalate to tiers 2 or 3.
Rinse and Repeat: Have Tier 2 personnel run a regular (every two to four weeks) refresher course for Tier 1 staff. How will you know if the training has been successful? Simple: If the Tier 2 personnel can truly rely on Tier 1 folks to troubleshoot while they're on vacation.
Enforce a Timeline: Being efficient means enforcing rigid timelines for fixing a problem or escalating it. "A lot of engineers, myself included, are like test pilots. They don't want to pull the ejection cord until they've almost hit the ground, and a lot of them die because of it," says Purser. So here's one possible timeline: "Tier 1, if you can't fix a problem in 15 minutes, then you need to call a Tier 2 guy. If they can't fix it in an hour, then you need to call in the Tier 3 guys." While Tier 3 may seem relatively expensive, so is lost productivity or sales.
Rotate Jobs: Don't pigeonhole Tier 1 personnel. "Just like in the NASA model, one person manages servers and data backup, another the fabric, another desktops, another networking. That's Monday. Tuesday, they come in and rotate jobs, so you never burn out, you never get stalled, so you get a good feel for all of them," says Purser. "Rotating positions and ensuring the people understand all of these things helps keep them fresh and ready to go, gives them experience, and in the end it makes our NOC more responsive."
Practice Coordination: Brush up on your ITIL, because creating really efficient operations requires overt coordination between IT operations, service desk and security. Leading practitioners often dub this as an IT command center (think: NOC on steroids). For example, Rich Hand, executive director of membership for HDI in Colorado Springs, Colo., recounts a recent tour of the Navy's command center in San Diego. "The three people who gave us the tour were the network operations manager, the security manager and the service desk manager, and they were all in sync." Operationally speaking, that's efficient.

Making network operations more efficient and responsive starts by ensuring even frontline NOC personnel have the right skills. Backed by training, clear operational guidelines and yes, a little respect, this is the start of what it takes to create a next-generation NOC.

The tools in today's network operations center (NOC) may be state of the art, but oftentimes the training, mentoring and overall approach need an update.

By Mathew Schwartz

<pOperations: The State of the Art

Build a Better NOC

How can organizations make their NOC more efficient, and better groom entry-level personnel for more advanced positions? Start with these five steps:

Sharpen Skills: Train Tier 1 personnel. If your company can't afford outside help, Purser recommends having your Tier 2 (specialists) and Tier 3 personnel ("the geek's geeks," in many cases outside consultants) design a training course. Just one week's training, he says, will increase uptime and reduce the number of problems that must escalate to tiers 2 or 3.
Rinse and Repeat: Have Tier 2 personnel run a regular (every two to four weeks) refresher course for Tier 1 staff. How will you know if the training has been successful? Simple: If the Tier 2 personnel can truly rely on Tier 1 folks to troubleshoot while they're on vacation.
Enforce a Timeline: Being efficient means enforcing rigid timelines for fixing a problem or escalating it. "A lot of engineers, myself included, are like test pilots. They don't want to pull the ejection cord until they've almost hit the ground, and a lot of them die because of it," says Purser. So here's one possible timeline: "Tier 1, if you can't fix a problem in 15 minutes, then you need to call a Tier 2 guy. If they can't fix it in an hour, then you need to call in the Tier 3 guys." While Tier 3 may seem relatively expensive, so is lost productivity or sales.
Rotate Jobs: Don't pigeonhole Tier 1 personnel. "Just like in the NASA model, one person manages servers and data backup, another the fabric, another desktops, another networking. That's Monday. Tuesday, they come in and rotate jobs, so you never burn out, you never get stalled, so you get a good feel for all of them," says Purser. "Rotating positions and ensuring the people understand all of these things helps keep them fresh and ready to go, gives them experience, and in the end it makes our NOC more responsive."
Practice Coordination: Brush up on your ITIL, because creating really efficient operations requires overt coordination between IT operations, service desk and security. Leading practitioners often dub this as an IT command center (think: NOC on steroids). For example, Rich Hand, executive director of membership for HDI in Colorado Springs, Colo., recounts a recent tour of the Navy's command center in San Diego. "The three people who gave us the tour were the network operations manager, the security manager and the service desk manager, and they were all in sync." Operationally speaking, that's efficient.