Getting People to Talk: An Ethnography & Interviewing Primer is a wonderful video made by students at the Illinois Institute of Design. It’s also a great example of a Participatory Video approach as one of the creators, Gabriel Biller, notes on his blog:
… we hope that this will be just the first of many videos produced by ID students to help teach others about techniques, methods, and other aspects of design through the powerful and entertaining medium of video.
I won’t try and summarise the film here other than to say that it does an excellent job of demystifying ethnography and showing the benefits of getting in to the field.
A few years ago I read an article that changed the way I do usability testing. In the June 2004 Gotoreport Erik Burns introduced café usability testing: recruiting and running usability tests with participants in local cafés. This was a revelation to me. Even though I was using ‘discount’ methods and didn’t maintain a formal lab, Erik’s method of selecting participants in situ offered the opportunity to streamline the whole recruitment process. I could evaluate designs more quickly and economically, and pass the savings on to colleagues and clients. I ran my first café usability session in 2005 and I’ve been hooked ever since.
If café usability’s new to you and it sounds like a useful technique, then here are a few tips to get you started and hopefully avoid some of the common pitfalls.
You can run café usability sessions anywhere. You don’t need to restrict yourself to cafés. Conferences, trade shows, events, museums, canteens, showrooms, student unions, user groups; wherever you think you’ll find people who match your target audience.
The bigger your recruitment sign, the better. In café evaluations you’re recruiting participants on the spot. If you’re on your own, your main tool is usually a sign offering an incentive. A4 desk signs are OK in public spaces where you need something inviting yet unobtrusive, but if you really want people’s attention you can’t beat a big poster. Position yourself next to a wall and stick the poster above your table (provided you’ve got permission, of course). Your sign does all the hard work and you can focus on the evaluations.
Get creative with your incentives. A big poster means you’ll get people’s attention, so now you need to concentrate on how you’re going to entice them to participate: the incentives. There’s less rigmarole for participants in café evaluations than in standard lab tests. You can be more adventurous and, um, budget conscious, in the kinds of incentives that you offer. The original article talked about free beer; I’ve offered chocolate, Gmail accounts, ice lollies, champagne and vouchers as well as the old workhorse hard cash. People tend to respond well to something a bit out of the ordinary, so go wild!
Position yourself where your participants are. This may seem obvious because you’re out in the field already, but your location in the chosen venue can really affect how many people you recruit. If possible try to visit the site beforehand to get a feel for busy times and places. Flexibility is the key; if you’re not seeing enough participants then it may be time to move.
If you’ve got help, recruit people away from your base. Take turns to go around the café/conference/canteen/hall/wherever and ask people if they’ve got a bit of spare time. Print some cards or simple paper leaflets with your company name, where you’re located in the venue and an outline of what you’re doing. Hand these out to people as you go around. At conferences, for example, there are often lulls early in the morning and late in the afternoon. Visiting other vendor’s stands and telling them about your evaluations can provide extra participants during these slow times.
Run your evaluations in public. I’ve seen some correspondence lately suggesting that you recruit people in public areas, but conduct the sessions somewhere private. Informality is the key to this technique. Running your sessions in public keeps it that way. Once you’ve withdrawn to another room you may as well be back in the lab. Nevermind that now you’ve got to manage participants in a separate room as well as maintaining your recruitment setup.
Bring spares of everything. In the lab all your supplies are usually to hand. In the field you’re dependent of what you (and any colleagues) can carry. You’d be surprised how easily simple things like mislaying pens or running out of paper can derail a session. So extra pens and paper are a must, but I’d also bring lots of consent and recording forms, copies of any paper prototypes (parts can easily get lost or damaged) and spare batteries for your laptop. And that’s just for starters. I’m sure there are lots of other things that you can think of for your specific situation.
Listen first, then ask. Café evaluations are conducted away from the lab, so this is a great opportunity to let your users lead and observe their behaviour. Try Mark Hurst’s listening lab approach rather than using pre-defined tasks. Talk to participants to discover what they would normally do on a website or application like yours. Note their answers and then ask them to try some of these tasks with whatever artefacts you’ve got (full site on a laptop, prototypes etc.). If there are specific areas that you want to test, throw in a few exercises of your own, but leave these until you’ve had a chance to see the participant’s usual behaviour.
Allow for more time, but don’t count on it. I usually ask for 20 minutes of people’s time, but the informality of cafe evaluations is infectious. Once participants have started they’ll often give you far longer. Have a few standby tasks or questions ready to make the most of each session. As with all research though, respect for the participants is paramount and that extends to their time. If you’ve asked for 20 minutes then that’s what you should aim for. (You’ll be surprised at how much you can achieve in just that short period.)
The 10th tip? Well that’s up to you. If you’re already running café usability sessions then I’d love to hear your favourite piece of advice. How do you do it? What do you do differently? If you’re just getting started with café evaluations then let me know how you get on. What did I overlook and what would you add? I’m sure that you’ll find the informality and freedom of café usability as stimulating as I do, even without the coffee.
This post originally appeared, in a slightly modified form, as an article in issue 75 of Interfaces, the magazine of the BCS Interaction group.
Dan Saffer asks about new techniques to try in 2007. (We’re closer to 2008 now, but I started this in response to Dan’s post so I thought I ought to finish it off before we hit New Year again.) Participatory Video (PV) has been around for a long time, but with YouTube making national news and video mobiles well established, a cut-down version could provide a useful addition to the toolbox of methods available to researchers and designers.
PV creates a story told by users in their own way about issues that matter to them. In the traditional version, participants film and share short videos. The researcher acts as a facilitator providing training to use the video equipment; a series of games to introduce specialist concepts like storyboarding; and help to identify issues for study. When done well, it presents the ‘inside view’ in a lively way that is accessible to people at all levels. It promotes the skills of filmmaking and storytelling and delivers outcomes that can be used as tools for education. This isn’t like a diary study, capturing things exactly as they happen when they happen. It’s about people telling a story, distilling the essence of an issue that matters to them. Remember a group project you did at school or college, then add video and you’re getting there.
If this sounds interesting, why not check out Insight UK’s fantastic ‘Insights into Participatory Video: A Handbook for the Field‘ (you can download a free PDF version of the book from their website; you just need to register a few details). It’s an excellent guide to using Participatory Video and a great example of an accessible and practitioner-focussed resource. Even if you don’t think there’s much mileage in PV I’d still suggest having a look at the handbook. It really is excellent. (I’m not in any way associated with Insight UK, I just think they’re doing good work.) And anyway, it looks like Participatory Video is already happening on sites like YouTube. The question is whether we as designers want to embrace it.
If you’ve read more than just this post (hurrah for The Subscriber), you’ll know that I believe a little research can go a long way. I also believe in democratising research, especially in team settings, so that everyone can learn. This story in the Daily Mail shows that anyone can get into the field and start learning about their customers.
Abramovich’s wife goes undercover as she learns how to run a hotel chain
Irina recently paid a number of visits to the chic Mayfair establishment Brown’s…, not as a guest - but as an undercover trainee. The elegant Russian has embarked on secret ‘work experience’ at the five-star hotel.
You don’t have to be a trained user researcher or a billionairess to live in your user’s shoes (although either helps), you just have to be willing to get out there and meet with them; to be open to what they’ve got to teach you.
There’s a useful new twist on an old technique over at frog’s Design Mind. In their digital diary study participants used voicemail, e-mail and digital cameras to record their behaviour. These insights were sent daily by e-mail and free phone numbers to researchers who followed-up through the same digital technologies. Frog believe that this rapid feedback created a conversational dialogue and a more responsive study. According to them, the technique is:
Efficient (digital methodology and remote facilitation require no travel)
Rapid (digital data collection accelerates response)
Scalable (process allows for parallel investigations in many countries)
Dynamic (immediate communication permits follow-up questions)
I can certainly see the advantages, and use of the technology would fit perfectly with certain groups like teenagers. But whilst Frog’s view is that the application of ‘digital’ facilitated the more conversational/ethnographic approach …
Feedback fostered a dialogue between researchers and subjects that mirrored true ethnographic interviews…. Traditional diaries allow for no interaction between researchers and subjects…. Data is restricted in one direction.
… I would argue that there’s no reason why paper-based studies can’t include a dialogue with the participants.
The paper form that I created for my student technology study. Participants reported and categorised their activities throughout the day. The form was based on the work of John Reiman.
I was in regular communication with participants when I used diaries to investigate the way students integrate technology and studying into their lives. I met with the students to brief them about the study and then followed-up each week with a face-to-face or telephone interview. Participants were given disposable cameras, alongside paper forms, to help document their behaviour and surroundings (this was before camera-phones or cheap digital cameras had become widely available). E-mails and texts helped to arrange formal feedback sessions and provide informal support between meetings. Not quite the information deluge that the Frog Design team solicited, but enough to keep this researcher busy in-between.
Digital diaries are definitely an exciting and agile method, but with many technology-driven approaches the technology itself can often become a barrier. I think there’s still a lot of scope for mixing the digital and traditional.
Carson Systems seem to be the people talking most sense about the recent upsurge in web application development. True to their ‘warts-and-all’ ethos they’re recording the development of their new product over at barenakedapp.com. Going on past form, there’s bound to be plenty of useful advice for budding web entrepreneurs as well as the odd pitfall to avoid. It’s brave doing anything in public view so hats off to them (although that’s as naked as I’m getting).
In one of those sublime serendipitous moments, I was invited to take the UK Internet User Monitor survey by another website just as I was finishing the previous post. It’s a great example of a long questionnaire that’s a data gathering marathon. Needless to say, I didn’t complete it.
Although the authors claimed it would only take 15 minutes, I’d already answered about 15 questions when I encountered this beast –
– and bailed out. Note that I’m only 20% of the way through and am being presented with 21 questions, each with several possible combinations. And completing this page took the progress bar to 22%!
True to their philosophy , the guys at 37 Signals provide a great example of a short survey that packs a punch. I like their, ‘What one thing would you remove?’ and ‘What one thing would you add?’ questions (although the actual phrasing they use is a little confusing).
It’s not difficult to remember. ‘Shorter is better. Shorter is better.’ Less time, less data, more responses, more action.
My experience with Mori got me thinking about how I use surveys to gather user data. Questionnaires are a great tool for providing a range of information. They can also be a useful adjunct to activities using small samples, such as observations or usability evaluations, where clients may be sceptical of the wider applicability of the findings.
I like to use questionnaires to provide context at the beginning of a project. The results give you a baseline and you can re-use many of the questions during usability evaluations to give continuity to your data. For ongoing product development, I’ve found that certain types of questions provide the most useful results.
Likes and dislikes
What do you like most about using <product>?
What do you dislike most about using <product>?
These questions can give you some very visceral responses. You’ll get tactical information about particular features - ‘I hate that it doesn’t remember my information from the Details screen’ - and strategic replies that reveal more about underlying motivation and usage - ‘It lets me keep on top of my appointments and manage my time’.
Product use
How often do you do the following activities with <product>?
Daily
Weekly
Monthly
Rarely
Never
Create a new page
Edit an existing page
etc.
These responses can help you determine where to focus your development effort. If something’s used rarely or never then perhaps it’s time to think about dropping it. Or maybe, if it’s important functionality, you might need some qualitative research to find out why it’s not being used.
Product feelings
For each statement please indicate to the level to which you agree or disagree.
Strongly agree
Tend to agree
Tend to disagree
Strongly disagree
I thought that <product> was slow
Learning how to use <product> was easy
<Product> responded quickly to my actions
Here, you’ll get an understanding of how people see your product and the match between your view of it and your customers’. Some standard statements you might want to include are ease of use and ease of learning. If there are key features or benefits that you’re using to sell your offering, put those in here too and see how they fare.
Whatever questions you settle on, shorter is definitely better for surveys. (This is especially true on the web, where random sampling means you’ll usually be interrupting someone engaged in a totally different task.) People are more likely to answer a short questionnaire (unless you’re providing an incentive), but there’s also the amount of data you generate to consider. You’re going to have to analyse all the responses from that 100 question monster. So shorter means you can discover the trends more quickly and act on them sooner.
If that’s piqued your interest, the following books have loads more advice:
After years of wondering who on earth they get to participate in those polls reported in the media, the other night I found out. People like me. As someone who’s designed, delivered and interpreted plenty of questionnaires it was interesting to see how an international company like Mori does it.
There were quite a few questions (the session lasted around half an hour), many accompanied by response cards with the various options available for each question. And to add that twenty-first-century touch to the experience, all my answers were noted down in a custom bit of software on a small Tablet PC.
A lot of it was standard fare: ‘What do you think of the following statements?’, ‘Have you ever done x before?’, but I thought some of the questions were definitely suspect. For example, from memory: ‘What do you think are the main things people look for when choosing a school for their child?’. Note that it wasn’t ‘What are the main things you look for when choosing a school for your child?’. It’s a subtle, but important difference. The first version is designed to elicit a specific set of responses drawn from ‘popular’ influences. Think Daily Mail headlines, school league tables etc. It’s a question looking to reinforce known answers and it’s bad practice.
So, even though it’s people like me participating in these surveys, poor question design means our answers aren’t always what we want them to be.
I’ve used PowerPoint in the past to spruce up wireframes with simple clicking and branching by exporting Illustrator files to gifs, importing them into PowerPoint and then overlaying buttons to jump to different slides showing different screens or states. I’ve also seen it used on other teams with much more finished visuals to good effect too. It’s a quick and effective method of getting screen-based interactivity that you can show to test participants, key stakeholders and other team members to evaluate and communicate your designs (and prototyping the new version of Office with the old version is so meta it hurts!).
