I've always been a proponent of leveraging usability test artifacts (video, transcripts, quotes) to help communicate to decision makers the impact of design decisions. I always do this with great care, since I don't want to overstate issues I've uncovered, while at the same time making clear the human effects of software usability. Recently, Microsoft partnered with a branding agency to run what appear to be usability tests of its Vista operating system. The real goal was to counter the the large amount of negative press around Vista. I'm a little concerned that what Bradley and Montgomery has done may damage user experience and usability professionals do by trivializing research and making user look like fools.
- Turns out that these were just 10-minute demonstrations by experts, not actual usability tests.
- Apparently none of the participants were current Vista users, so their ratings of Vista at the start were entirely perception-based. Unsurprisingly, their approval ratings were astronomical.
- This is represented as scientific by calling it an "experiment", and using candid-looking video clips of users in a lab setting. Video clips are not shown in their entirety, nor is a transcript of the interview made available. I didn't expect otherwise, since even if this was real product research, this stuff wouldn't be made public.
- From a methodology standpoint, they out-and-out lied to the users about what the software is and what their intentions were. To make matters worse, "gotcha" clips are shown of users after it is revealed that they are indeed using Vista.
- I don't have access to the genesis of this project, but its clear this was not an open-ended inquiry into the usability of Vista, but a marketing exercise from the outset to prove that Vista is fine and there's some unjustified bad press out there.
Is this going to hurt user researchers? I doubt it, given how few people will view the marketing site, but it is a real danger. I wonder how the product folks at Microsoft are viewing this.
*Update: The response at the MS Vista product blog isn't entirely positive either.