A presidential campaign is many things to many people: a reason to hope in the future, a wellspring of jokes and debate fodder, an annoyance to tune out, a chance to participate in the civic process. But for a couple dozen software engineers and developers involved over the past two years in President Obama’s re-election effort, a campaign was something entirely different: a billion-dollar tech startup with an eighteen-month lifespan and a mandate to ship code under extreme pressure. Speaking recently to a New York City audience, some of Obama for America’s leading tech people—those involved in the all-important Dashboard and Narwhal projects, as well as fundraising and DevOps—cheerfully characterized the experience as “insane,” filled with unending problems and the knowledge that, at the end of the whole process, nearly everything they worked on would likely end up tossed away. Nonetheless, they seemed proud of how much the Obama campaign bet on technology. “The campaign realized there was enormous value in reaching out via new media,” said Jason Kunesh, Obama for America’s director of user experience. “When it came to the 2012 campaign, [we took it] a step forward and had an in-house engineering culture.” In addition to reaching out to voters through social-networking channels such as Facebook, the Obama campaign (along with the rival Romney campaign) needed to data-mine the nation for votes and donor dollars. That meant hiring dozens of workers skilled in everything from building apps to predictive analytics. And it meant depending on complex systems and technology untested at such epic scales—something that would eventually affect one of the two competing campaigns in a very public, negative way.

“Don’t [Expletive] This Up”

The Obama campaign had a preference for hiring senior developers, which meant nearly everyone on the team had an extensive skillset. They worked in weeklong sprints, with the goal of releasing new software every week. They were determined to remain technology agnostic, building apps in a variety of languages—Ruby on Rails, Java, PHP backed by Kohana. The management and engineering teams worked in close physical proximity, helping close the feedback loop: whenever something went drastically wrong, the path to fixing it started with a ten-yard walk to another cubicle and smacking the responsible party in the back of the head. In addition to roughly 40 tech people in the “core” group, legions of volunteers also contributed their time and skills. Early in the ramping-up, that core decided to coalesce software development around product teams. The Dashboard Tool, which the Obama campaign used for volunteer organizing and metrics, involved everyone from designers and analytics gurus to DevOps. That allowed the managers to break workers out of silos, the better to let ideas flow back and forth. The campaign also got to work on Project Narwhal, an attempt to take all the potential voter information accumulated since the 2008 campaign and bring it together on a single platform, accessible to the growing plethora of campaign-related apps. “Even the voter-protection stuff we did, as the election got closer, was through Narwhal,” said Ryan Kolak, lead engineer of Narwhal Integration. “Because of its size, we had to choose which problems to solve.” Just to add to the fun, they were also starting the campaign’s biggest tech initiative from scratch, with the exception of some already-built vendor APIs. “We went in with a complete green-field,” said Chris Gansen, the campaign’s Dashboard Tech Lead. “We had to build Narwhal while we built the apps on top of it.” That many cooks, of course, raised the specter of the whole kitchen burning down. “Everybody who was inside headquarters had a viewpoint,” Kunesh said. With regard to field offices, “each one of the fifty states did things differently.” With months to go before Election Day, the tech group set up a big board at campaign headquarters that diagrammed the various initiatives underway. That brought in a lot of input from other campaign workers, some of it valuable; but the tech people still had to overcome an innate reluctance on the broader campaign’s part to embrace technology. “By showing it works, we overcame resistance to the iterative building process,” Gansen said. “We had a massive roadmap,” he added. “We tried to winnow it down: what can we do that the vendors can’t do, that would make people’s lives easier? We swung wide and built this really big, bloated application—and then chopped off the things that didn’t work very well.” The campaign depended increasingly on technology for everything from fundraising to voter registration, ratcheting up the pressure on the developers and engineers. They knew that a loss of focus could have much broader repercussions: at Obama headquarters, the software group ordered a cake with the words “Don’t [Expletive] This Up” written across the top (note: the actual cake contained the profanity). Nobody touched it for quite some time, as if slicing into that sugary deliciousness would send bad luck the campaign’s way. And as Election Day approached, it appeared that Obama’s tech people would need an extra-hefty dose of good luck: rumor suggested that the Romney campaign, strengthened by millions of dollars in donations, was prepping a software weapon of its very own.

Orca

Romney’s people were hard at work on a Web-based app that would allow swing-state volunteers to update Romney headquarters in real time on Election Day. Armed with continually refreshed lists of who’d already voted, headquarters and various field offices would then make the necessary calls to draw more voters to local polling places. In a close election, a few hundred voters in a fiercely contested swing state could mean the difference between bitter defeat and narrow victory. The Romney campaign even named the software “Orca,” after the whales that chow down on tasty Narwhals. (Never mind that the Obama campaign’s Narwhal was actually the platform backing a collection of apps, not the app itself.) But on Election Day, Orca melted down in spectacular fashion. Field volunteers reportedly received no training in the software until Election Eve, when the Romney campaign emailed them a massive instruction manual and voter-roll file in PDF format. According to a postmortem by Ars Technica, Romney’s IT crew decided to support Orca with a single Web server and a single application server—hardly the sort of infrastructure backend capable of handling the data tsunami of Election Day. Orca’s problems weren’t limited to paltry infrastructure. “I worked on the Colorado team, and we were called by hundreds (or more) volunteers who couldn’t use the app or the backup phone system,” one unnamed source within the Romney campaign told Breitbart soon after the election. “The usernames and passwords were wrong, but the reset password tool didn’t work, and we couldn’t change phone PINs. We were told the problems were limited and asked to project confidence, have people use pencil and paper, and try to submit again later.” Untrained Romney volunteers also tried to access the Orca Web app by typing “http” before the URL, instead of the “https” needed for a secure connection. Since Romney’s tech people hadn’t coded to auto-forward “http” to “https,” those volunteers landed on blank pages. Many abandoned Orca out of sheer frustration. In a perfect world, the Romney campaign would have led its volunteers through several dry runs on Orca before Election Day; but someone had decided to keep the major parts of the software a secret for as long as possible. As a result, Orca proved toothless against Narwhal and the other Obama apps. The Obama campaign, by contrast, knew that volunteers and field offices would struggle with the technology. The tech people concentrated on making the systems as simple as possible, and used early voting periods in some states to test software. They had built each piece of IT infrastructure to fail, relying on Amazon’s Simple Storage Service (S3) for multi-region redundancy. But the possibilities still kept them up at night: What if the multiple payment processors failed simultaneously? What if an Amazon region went down unexpectedly? What if the fundraising pages stopped working at a crucial moment, such as after the second debate? As Election Day neared, they also found themselves confronting an unexpected issue: Hurricane Sandy. With the storm rumbling toward the East Coast, the campaign made the decision to shift the bulk of its IT infrastructure from Amazon’s hosting data center in Virginia—a little too close to possible landfall for comfort—to one in Oregon. The whole job took 24 hours. “Ten years ago, that wouldn’t even have been possible,” one of the campaign’s tech people admitted. So on Election Day itself, the Obama campaign’s IT infrastructure ran smoothly. “I was the tech liaison in the boiler room,” Kunesh said, a period he characterized as “boring.” Obama won the election by a safe margin. America’s attention shifted from electoral politics to the upcoming holidays. Someone finally sliced into the “Don’t [Expletive] This Up” cake. “We had faith in our tools on Election Day,” Gansen said.

The Future

The campaign’s software builders believe their type of work will become more prevalent in politics as time goes on. “You look at the more revolutionary stuff we did, registering people to vote through Facebook and so on,” Kunesh said. “That’s only going to increase.” Obama for America’s success with those software tools may also convince future campaigns to embrace similar initiatives. However, presidential candidates in 2016 won’t have some of Obama’s advantages. With no primary challenger, and an extensive network of donors, his campaign had the cash and time to focus on building out its backend infrastructure. Four years from now, a whole new round of presidential startups will emerge, doing their best to transform data into votes. Decades ago, campaigns elected their candidates to the Oval Office with what we’d consider a bare minimum of technology by today's standards: voter rolls printed on paper, banks of telephones manned by volunteers. In 2016, by contrast, extreme technology such as apps and advanced analytics could prove the key to victory.