IGF and Human Judges: An interview with Jenn Frank

In the latest controversy to come out of the IGF judging process, Unwinnable writer and freelancer extraordinaire Jenn Frank seems to have struck a nerve with her willingness to show that there, in fact, humans behind the judging of the IGF games.

Over a period of several days, she was nice enough to field some questions about being a judge, what the process is like and what we can do to support the IGF going into the future.

How did you get picked to be an IGF judge? Is there a ceremony and some ritual sacrifice… or do they just pick people they know to be articulate about games?

Yeah, it’s kind of unceremonious. It’s really only an email inviting you to volunteer your time, and could you please also list what systems you have lying around. So you’ve already been vetted in some way, and now you are being invited to commit.

I can’t actually tell you how they choose the IGF’s judges, because I don’t really know. But as far as I can tell, it’s some cross-section of already-established critics and developers. From what I’ve observed, it’s a pretty good spread of critics and developers who are trying to score these games.

And obviously that’s great, because developers and professional critics are going to interpret games in these totally different and disparate ways, so if you can get so many developers and so many critics judging a single game, the game can be gauged more objectively.

It’s a really strange balancing act, though. It’s almost like trying to assign seating at a dinner party.

I should tell you, I’m pretty sure I’ve only been a first-round judge, which is to say that I am probably only being used to weed the broken games from the playable.

How do you judge games for the IGF? Is there a criteria, do you make your own or do you just switch it up per game?

So there’s this online portal we all use. It’s a really important tool. I do think we have to talk about the actual tool, if we’re going to talk earnestly about the process.

Every game has its own individual hub, with the developers’ gameplay instructions, installation instructions, maybe a trailer video that I’ll never watch. There’s a link to the FTP server, where the judges can access all the builds of the game, and then there’s a forum at the bottom of the page where judges compare notes. And then there are these fields for the actual scoring.

Now, carelessly and without peeking at previous years’ judging portals, I can tell you it keeps shifting slightly, I think to constantly evolve with suggestions as to how the system can be improved. The last year I judged, it had been simplified somewhat: I think it was, instead of leaving scores based on several different virtues, like sound or art or whatever, I could give any one game a “single score” but also nominate whatever game I wanted to nominate for “best sound,” “best art,” “best game on an iPhone,” “the Nuovo Award.”

At this online portal, every game also has a feedback forum. The judges are all accountable to one another, if I’m not mistaken — you can see all the other judges assigned to any one game, I think? — and judges are able to announce in these forums which games aren’t installing, where they’re hitting glitches, is it like that for the rest of you guys.

Sometimes the judges also discuss how they’re actually scoring, and while I admire that style of collaboration, I really wouldn’t want to share my actual personal scoring rubric. Everyone has his own scoring rubric, I have my own bases and bias for what makes a game work, and that’s just personal, unique to the critic, yeah, absolutely. At the magazine I reviewed for we actually had a policy about not discussing scores, in the interest of not influencing other critics’ reviews.

But there are always standards, like this number is for “great,” this number is for “so-so.” So while personal criteria might differ, sure, there’s an idea in place of what precise numerical score means a game is good or bad.

I think I prefer the “single score” idea because we’ve all generally shifted to metascoring anyway, like, as a culture. But also — coming back to the idea of the “personal rubric” — sound isn’t always that important to me, or art isn’t always important, or whatever, because I’m always trying to grok at some underlying mechanic or philosophy. So my continuum of “good” is probably different from the next judge’s, and I do prefer the idea of having each critic sort of assign his own metascore, just because there are some things you shouldn’t bother trying to standardize. Like, telling judges HOW to score is bogus — you’ll end up with just one type of judge, and then the scores won’t be accurate at all.

Optionally, a judge can leave his reactions for each game, to eventually be read by the developer. I always try to say something about each game, no matter how long I’ve spent playing it, just because I want to talk to developers about my issues, or what I liked and what ought to be kept. Constructive criticism is the most important part to me; otherwise, what’s the point?

Then again I was a fiction writing major in undergrad, so I am coming at game design from this place where you always workshop something, no matter how good or bad or how much potential it has. Like, even if you can’t suggest any concrete edits, you always conclude with at least a paragraph about what was good, what didn’t work for you. In the case of IGF these comments are supposed to be kept anonymous, which is only fair to the judges, because it gives professional colleagues an opportunity to not hold their criticisms back. That said, I am pretty sure I am being truthful when I report that I “sign my shit,” because a developer might as well know whose opinion it is and decide later on whether to take my opinion for whatever it’s worth.

Once you’re finished with your little crop of games you’ve been assigned, you’re actually allowed to freely roam the archive of games and play and score those, too. So a game that has already generated its own buzz might have a lot more people investigating it, scoring it. So it’s true that a game with more positive judge-buzz will have this beneficial feedback loop, and the game eventually gets more scores than it really even ever needed or merited, just because of a consensual interest.

I won’t say that’s bad, because I don’t think it is. It’s the same idea as Kickstarter or any other crowd-sourced method of judging a thing. Like, maybe certain projects on Kickstarter fall to the bottom and don’t get funded and I disagree personally, but when the crowd decides to very unilaterally and democratically back one thing, I just kind of go, “OK.”

Part of the frustration, I felt, coming out of your Unwinnable post was trying to play games that wouldn’t work for some reason or another. Is that common? My perception of the process, through your post and people talking about it in the past, is that there are a handful of games each year that many judges try to play but are unsuccessful. Is that accurate?

I really don’t know whether it’s common because, before my rant at Unwinnable, I’d never openly discussed judging IGF, nor had I followed up on reading about other experiences. In my rant’s case, I was just trying to sincerely describe a personal judging experience. I still don’t know how common this phenomenon is, even, because a lot of public forum comments told me I was so, so wrong, while a handful of private comments from ex-judges told me I was right. So I don’t even know how to interpret that evidence.

In my very personalized experience, it can be difficult to make a game run without first installing any number of new libraries on your mom’s brand new, bleeding-edge computer. Now, I totally pride myself on being able to do that sort of thing because I am a geek from way back, but when a developer gives me an installation file that can automate the process, I really appreciate it, and I am already taking that luxury into account. Like, thanks! Thanks for doing that, so much! I’m not saying you have to defer to a judge and really respect his time and resources like that, I guess, but when you do, it makes it so much easier for your work to be judged honestly and fairly.

I did run into broken games a lot. Like, often enough to be a huge chore. I think I’m really patient, generally, and if I have enough time, which, I usually do, I’ll go that mile to finagle libraries and settings and “run as administrator” or “run in 32-bit” or “run in a different resolution,” tweaking anything until something works. But sometimes something simply won’t run, and I feel really deep, exhausting guilt in sidestepping that game. I’m also pretty open about being a procrastinator, which is my politest way of saying, if I spend more than an hour trying to make your game work, I have to take it on the chin and move along.

One anonymous commenter did tell me he’d “literally” never had an issue running a game, and I had to wonder what planet he was from. Seriously, what supercomputer does he have, that everything will run unborked? It really just made me feel kind of poor. Like, maybe judging is only for the rich? Maybe you really should only judge if you have this incredible device that can run everything no matter what. But that’s a really insular attitude. I got into indies, as a lot of people do, because I was into freebies and handouts. That’s because, by any definition of class status, I’m really, really poor. And maybe that’s the problem here, with my ability to install and run games and review them? It’s a really scary suggestion to face, but who knows? Maybe being allowed to judge a game really should be more like being allowed to rent an apartment. Like, go ahead, run my credit score, assess my finances, see whether I can afford to run your game. I don’t know.

Was it common to see multiple versions of games. You mentioned, for example, “computer-sciencing” your way into games to play them. Was that a constant problem or just something that crept up from time to time?

Definitely. Someone hinted at it elsewhere, but developers are afforded absolutely every opportunity to refurbish their builds, it seems. I’ve seen new builds appear even after judging has formally ended. And whatever, that’s great! When I write, I go, “Whoops! Here’s my real final draft, for sure,” and then I’ll do that twelve more times to my editor, because I’m constantly finding these imperfections and tweaking them. Like, I totally get that mindset. You want to submit your best, and that’s wonderful.

But when I am having trouble getting a PC game to load, say, and I’m staring at an FTP server with fifteen files with all these weird names, I get to this point where I confidently download all the files with the most recent timestamps, I install all of them, and things still won’t work, and I have no instructions, and it’s all very frightening. What the hell am I installing right now? And so I have arrived at this moment where, because I am a human judge and not a perfect robot, I concede defeat. I really have to. Because the rub is, if I don’t cut my losses and move on, I am doing an enormous disservice to any game that remains to be played.

And sure, I’ll try to come back to yours to reinstall it again, but when I was doing all this in 2010, I did finally hit a wall where I was just, I’ve never had a better computer in my life, and I have like three games that I can’t run. I keep trying, and I’m done trying because time is up.

And I really don’t think this is anybody’s fault, exactly? A game didn’t have quite enough user feedback before it was submitted; it happens.

But I would not launch a website before testing it in every browser, on every system, in every resolution, because that is not what I would ever want to put out there for everyone to see. I’d want to make sure first. And in keeping with this terribly stupid metaphor, if my most major complaint is that my new website has a serious “bounce rate” or whatever, I wouldn’t yell at my own visitors. I’d want to examine what I can fix.

Again, this analogy is pretty much the worst, and it especially fails when you bring that $100 entry fee into it.

But the judges themselves are very deeply invested, and it’s because they aren’t being paid anything. I’ve read some suggestions that the IGF could try paying its judges, but no, the IGF shouldn’t: there’s a big socio-psychological tenet behind this thought, because judges will actually give more if they get less out of it. Like, as an outsider with only a passing interest in sociology, this is a real fact. You literally cannot pay a volunteer to do a better job.

As a way to wrap this up and let you get back to enjoying GDC, what can bloggers, podcasters or just enthusiasts about indie games do to support the IGF?

I just got out of a screening of ‘Indie Game: the Movie,’ and just watching it did a whole lot to evaporate whatever cynicism I’d lapsed into feeling. Like, there are real humans, and real human battles, behind every one of these games. And this documentary film goes to great lengths to frame what a gut-wrenching, all-or-nothing, everything-on-the-line process it is whenever an individual tries to produce anything of moment and value.

But in the end the work will always have to speak for itself, and — I’m confident about this — if the work is any good, it’ll be a success. Or it will at least be acknowledged in some way.

To see a movie like this screened, like in an actual auditorium or venue, you’re kind of awestruck by the magnitude, the total outpouring, of real love and support for indie developers. It’s very moving, and witnessing the audience’s reactions to the movie was almost as inspiring as the movie itself.

Honestly, I’m not sure there is a more human industry than indie game development, and when other people love and support and write about or buy these games, people are also acknowledging the humans behind the games, their work, their efforts.

A lot of work and effort goes into “loving” and “supporting,” too. It’s easy for anyone to burn out, to get exhausted, no matter what role you play in this process. Even “games criticism” is this extremely human, subjective process — it can’t be automated. None of this can be automated.

You asked me how bloggers, podcasters, and “indie enthusiasts” can keep supporting indies or the IGF or whatever. And I don’t have an answer? Or I do, but it’s really very twee: keep loving. Don’t get tired of being totally in love with everyone and everyone’s work. That’s all any of us can do.