I was dismayed when the new IGDA leadership started off by announcing a member survey. What’s the big deal? Well, to put it bluntly, surveys are lame.
First of all, it rings of amateurism. It’s one of those bright ideas you expect from fresh-faced business school grads, along with “branding” and “monetization”. (“hey, let’s do a survey!”) And I’m not saying this is something just biz-school grads do – I joined a engineering-oriented startup just before they sent out a survey to potential customers and frankly, it was a waste of time. Everyone has their pet not-well-thought-out idea of how to design a product – if they really know what they’re talking about, you should hire them or work for them. And most of those surveyed weren’t the purchase decision makers, anyway.
Second, it’s annoying. I get pop-up surveys every time I leave my online bank account (“What can we do better?” “Get rid of this pop-up”) and other sites (like the KTLA news site). And I’ve only seen one or two on-line surveys that bothered to tell you how many questions are included up front are provide a progress indicator. The last IGDA survey I started said “this will take approximately ten minutes” – that’s both vague and unappealing. I once got suckered into taking a survey at GDC to get the “free” t-shirt, but after ten questions I asked the attendant how much longer it would take. I forget the answer, but it explained why the line was so long, so I bailed out. At least on-line surveys are less obnoxious than telephone surveys (“this will take just a few minutes…”). Non-profits are guilty, too – before I got rid of my land line I used to receive telephone surveys from non-profits (“help us out with our study”). I even left one non-profit after I showed up and they put me on a phone to call donors (“thanks for donating, so now I’m going to call you at dinnertime…”)
Thirdly, surveys generally lack rigor. You can construct and interpret a survey any which say – it’s like a focus group without the free food. And it’s the laziest form of research, like the students who post their homework questions online and ask for answers. There is some entertainment value, like the call-in opinions solicited by news shows (although I could go to a bar to get a man-on-the-street opinion, and a beer to go with it). I always check the Game Developer salary survey out of morbid curiosity, but I wouldn’t cite the results in a scientific paper. One year I was surprised to see the highest salary listed was $200,000 and then noticed they mentioned that they omitted any salaries over $200,000. I put much more stock in Christer Ericson’s article that lists real salaries based on Department of Labor records.
Finally, there’s no substitute for real data, but if you’re searching for subjective conclusions, swing the other way and really talk to people. In the startup mentioned above, there was an existing loyal following of customers who could have acted as an advisory panel. The IGDA already has an active member forum (and if it’s not active enough, there’s one thing that could be fixed – how about an RSS feed?) and personally, as someone who’s not only a paying IGDA member but a paying studio affiliate member, I would have appreciated something more than a mass e-mail soliciting my input. And sometimes, if you really want to see what the situation is, just look around. To pick on the IGDA one last time, I looked at their latest diversity survey. It’s well intentioned, and probably a good idea (like the Quality of Life survey), but studio owners and managers aren’t going to knock themselves out reporting how much they resemble a Minuteman patrol, and I’m going to guess your average white male isn’t going to be startled that he’s surrounded by other white males. But just browse around the forums where female game developers compare notes (it’s both horrific and entertaining) or just attend GDC and take a look around, especially in the progamming sessions. Even I was taken aback in a DirectX lecture when I looked around the hundred-plus crowd and couldn’t spot any women (not that I was expressly searching for some) and saw only a handful of non-white males. For real awareness, there’s nothing like immersion.
So, if you really think surveys are useful, well, just include that question in your next survey…