Big Brother is watching: how developers use game analytics

Information is a high-value commodity in the digital age. Retailers like Target and companies like Google are regularly on the lookout for information they can mine and analyze to produce effective, highly personalized products. The gaming industry is no different: games like Mass Effect and Halo monitor a player’s progress and send a bevy of information back to developers. In an effort to better understand how developers use this information, I spoke with Ben Medler, a visualization designer at EA who has worked on data analyzing tools for titles like Dead Space 2 and Star Wars: The Old Republic.

Analytics manifests itself in a variety of forms: before monitored playtime via customer-visible systems, companies also use playtests and market research. Much data is tracked, from how long it takes someone to finish a game, to what players prefer and how they falter – a far cry from the more dubious types of data tracking, like looking at a user’s credit cards or medical records. This tracking is done in an effort to better learn a player’s tastes and habits, ideally resulting products that can better serve players. Dragon Age 2’s introductory level, for instance, was a direct result of analytics. Bioware saw that a good number of people stopped playing Dragon Age: Origins after the first couple of hours, so it was clear that they needed to reconsider how to introduce future games.

Player data is also used by players, usually in competitive contexts. Call of Duty Elite, for instance, sells itself on the idea that it can help players improve their game by providing data to players. These systems can act as “dossiers” for players, and they’re built so that players can improve their playstyle, and also help them “show off their scores, to display their achievements and to basically say “I’m better than you.”

Game analytics is not a nascent field, Ben tells me, when you consider that the science of analysis has a history beyond that of games. And even within games, analytics has existed before joysticks: sports have tracked player data for a century, for instance. The devices we use to analyze the data, however, has gotten sophisticated over time thanks to digital tools. As a result, designers find themselves with a bevy of information they hope they can use to improve their games. “CCP hires economic theorists to run their analytics team, Valve employs psychology theorists to run player studies,” Ben says.

He cites Halo 2 as the first instance of a “large-scale, polished system” for data tracking. Microsoft has been a pioneer here, in that they conducted a lot of major research at the same time as Halo 2 for “dealing with how to setup proper player tracking systems for beta tests and for post-launch testing.” Cue the Xbox 360,  and the Kinect, both of which allow data to flow back to developers for analysis.

The degree to which analytics influences design is kind of murky, Ben remarks. We have two camps of people: those that feel that using data can hinder creativity and even worse, force them to “design to the lowest-common denominator,” and we have folks who find the data useful “during an iterative design process when lots of systems and features need testing.” There are benefits, and there are drawbacks, to be sure. “Having the ability to capture data from tens of thousands of beta testers, instead of solely relying on survey or forum post data, can be very beneficial when designers are testing their game,” Ben elaborates. Knowing how to use the information means knowing how the game functions, thus knowing how data is supposed to help you refine your systems. Otherwise, if you don’t know what you’re looking for, the information can’t be useful–it can even be a hindrance. Nonetheless, we always need to leave room for creativity, Ben warns. “If designers don’t take risks, go against the data results and try something new then they will be forever trapped following whatever the data says they should do next.”

Such is the case with some designers–Ben told me of someone at Google who couldn’t so much as change the color of a button without testing to see if it was popular with users. Google, of course, is famous for its obsession with even the most minute details of its design, and we’d be remiss to not attribute their wild success to that attention to detail.

Then we have companies like Zynga, which are infamous for their data-driven design. Despite incredible success, players are wary of Zynga’s approach in that there is fear that data is being used for ‘evil.’ These games are designed to tap into compulsion, hoping to keep the player engaged long enough that paying micro-transactions is an inevitability. To Ben, that’s what some data-tracking comes down to.  “I don’t think they have any evil intent, it’s just business. Facebook already looks at what games you are playing and for how long. They then use it to serve ads. Data will always be used to monetize the shit out of any service” Other companies rely on data heavily enough that players feel that games are losing their “soul,” that games are becoming worryingly mechanical. This is the conflict central to data driven vs intuition driven design. “Designers are often auteurs, they have a vision they need to express. Data-driven design is a threat to their vision,” remarks Ben.

Social games in particular aren’t designed quite like most other games, though. “They are built to be short, daily experiences which operate almost like webpages instead of video games,” reveals Ben. This means that A/B testing works better for these games, since “players have to deal with so many interface elements and smaller game worlds. Zynga can make ten different store interface windows available to ten different player populations but the gameplay is not hinder[ed].” Ben concedes that not every game can be as calculating as social games tend to be, because it wouldn’t always work. Even so, for games that do use this design method, developers can look toward web analytics and the lessons that people have learned in older implementations of data tracking. “We still hear [horror stories] from time to time, sometimes from places like Google or other times social gaming companies, but when you talk to those working in game analytics they mainly see themselves as providing an extra service or tool that designers can utilize.”

For other games, data tools can sometimes be problematic. When working on Dead Space 2, Ben developed a specialized tool called “Data Cracker.” This tool was highly personalized for the Dead Space 2 team, using both Dead Space artwork and color schemes. This approach is not typical; instead, there are a number of ‘general’ analytic tools that developers use. “General tools begets general analysis. Game developers spend years of their lives working on a game so to turn around and give them tools which can just as easily be used to analyze banking records is an insult to their craft.” remarks Ben.

Dead Space 2 is a good example of the battle between design intuition and data-driven design, actually. During development, the technical director created a “fun score” graph – a combination of a number of trackable stats as a means of visualizing how much fun a player was having. Needless to say, there was debate amongst the Dead Space team over what fun meant in the context of Dead Space 2 and to what degree the analytics could actually relay how much fun a player was having. This caused “a few designers to dive into the data themselves to see if they could come up with a better solution for tracking fun” In the end, Ben feels that this was a process that helped some developers grow from skeptics to believers when it came to analytics.

So where is game analytics going in the future? Ben thinks it’s inevitable for information visualization to become an even bigger part of game design just by proxy of how much information must be made available to a player. Data isn’t just being collected from players for players to use, though. Game analytics also provides huge benefits from a business standpoint. “Analytics can mitigate risk and protect the bottom line,” explains Ben. Services and systems like Halo Waypoint, The Cerberus Network, Rockstar’s Social club, SC2gears, Battlelog, COD Elite and Ridernet are just the tip of the iceberg.