Independent video game developers face plenty of challenges—but perhaps none take on a harder task than those building online multiplayer games. Creating a multiplayer game is “exponentially more complicated” than making a single-player title, according to developer Vince Fennel. But that hasn’t deterred Fennel and his team at Shield Break Studios from making Bierzerkers, a melee combat game in which teams of five players face off against each other. Fennel and Chris Talucci, formerly of Blizzard Entertainment and now co-founders of Shield Break in Costa Mesa, California, told Dice about the challenges of making an online multiplayer title and why they’re hosting the game on the Amazon Web Services cloud instead of on their own hardware. When players are competing against each other in real time from all around the world, “you’ve got a completely different technical problem to try to solve,” Talucci said. “Rather than just, how do I make this character look cool and fun,” a developer must consider “how does the other player feel like he's projecting himself into your computer at the same time that you’re reacting to that player. That’s a fun challenge of multiplayer games and why they're more challenging than just doing single player.” Hosting a multiplayer game requires a fair amount of computing resources, raising one important question: should an indie studio host its own servers, or go with a cloud service? Fennell and Talucci said they didn’t spend much time making that choice, as they saw little reason to operate their own servers. With Amazon, they can turn on new servers at any time without ever needing to touch a piece of hardware. “We knew we wanted to host as little architecture as possible locally,” Talucci said. “We're still that way, we really only run desktop computer development at the studio now.” While Shield Break hosts the game itself on Amazon, “that's not the only thing we're using [Amazon] for,” Fennel said. “We're also using Amazon for hosting our version control, we also use it for creating builds and then we distribute our builds through Steam… we could have somebody working anywhere and have just as much connectivity to all of our infrastructure as the guys who are working here in-house in the studio.” Shield Break generally pays 20 cents an hour for each virtual machine instance, though that can rise to $1 an hour for a build server with a decent graphics card. Shield Break lowers costs in part by keeping some virtual servers turned on for only part of each day. “We're not even in the thousands per month,” Fennel said. “It’s not a significant expense compared to operational stuff.” Shield Break has nine employees including Fennel and Talucci, and is just getting started with Bierzerkers. The game had an early access release on Steam last year, and version 1.0 is expected to be released Thursday this week. During its early release, Bierzerkers got up to about 700 players a day and 50 concurrent players at any given time, and will—the developers hope—scale up to far more than that after it exits the beta stage. To get ready, Shield Break has written custom code to automatically turn on new Amazon servers whenever more hardware is needed to host additional players. “It’s kind of scary, we'll see if the scaling really works,” Talucci said.

2 guys, 6 servers, 1 big game

If Shield Break hits it big, Fennel and Talucci wouldn’t be the first indie developers to make a widely played online multiplayer game. Growtopia for iOS and Android (with beta versions on Windows and Mac) is a massively multiplayer world-building game that regularly has more than 55,000 concurrent players. And it was created by just two developers, Seth Robinson and Mike Hommel. Robinson wrote about how he and Hommel built Growtopia in June 2013, when the game attracted just 3,000 players concurrently. Now, the game regularly hits 55,000 concurrent users and (so far) hosts 25 million user-generated “worlds.” Robinson recently gave Dice an update on the game’s technical operations. “Growtopia is running on five dedicated Intel Dual Xeon E5-2690 V2 boxes—each one has 20 cores and forty 3.0 GHz threads,” he said. “One is the SQL and logon server, one is purely world data file serving… and the other three handle hosting the active worlds instances.” A sixth dedicated server is used to run the game’s website and serve update files and world renderings generated by the game’s players. Instead of using a cloud service such as Amazon, the developers operate their own servers at the facilities of hosting providers Liquid Web and Servint. Growtopia cost just $99 a month to run when Robinson and Hommel started out. But with the game needing more infrastructure to support its current popularity, “our hosting costs are around $12,000 per month,” Robinson said. “We avoid signing any long-term plans and always stick to month-to-month to give us the flexibility to switch hosts if service degrades or an insurmountable technical limitation rears its head.” Since the game’s humble beginnings, the developers re-wrote the server code to take advantage of multi-core machines and “to shift players around separate machines to load balance,” Robinson said. Robinson added that he hasn’t used cloud-based server platforms, in part because “I'm afraid of anything I can't fully control or easily switch from if something goes wrong… I'm a control freak and want to know the brand and model of each piece of hardware and be able to replace anything, as a single bottleneck can ruin everything.” The setup hasn’t been problem-free, though. Occasionally, entire countries have been unable to play the game due to routing problems. “As we're stuck where we are geographically there isn't anything we can do but wait and hope,” Robinson said. In one case players from Singapore were unable to reach the server. “We could see the packets just disappeared at a certain spot between us,” he said. This wasn’t the data center provider’s fault, but “only having one place to connect to meant we couldn't give them any kind of alternate server address to play on.” With Amazon, developers can run servers in multiple countries to potentially avoid such problems, but doing so requires extra work. Running Growtopia is enough work as it is, although the developers have found ways to automate a lot of the technical tasks. “We've really gone out of the way to make the server self-running. It handles daily challenges, random events, and automatically chooses when special events like Circus or ‘Locke the traveling salesman’ come to town each month,” Robinson said, referring to certain in-game occurrences. “The goal has always been that if we forgot to check on it for a month or two it should still be running when we finally decided to show up.” Amazingly, Robinson and Hommel built the game without ever meeting in person. “He's in California and I'm in Japan,” Robinson explained in his 2013 article. For all the technical complexity of running an online multiplayer game, it turns out that managing the people who play it might be the most difficult job. Growtopia has about 18 volunteer moderators who help manage the community and forums, and Hommel told Dice that developers building online multiplayer games should “be prepared to discover that upwards of 75 percent of your workload is going to be community management.” The Growtopia developers worked to improve systems for moderating behavior and banning players, and “putting notes on accounts to keep track of things players have done,” he continued. World-building can be a pretty contentious exercise, it turns out. Part of the job, Hommel said, is keeping players from “murdering” each other. In some cases, he means that literally: “Death threats are a daily issue.”