At 09:06 AM 3/29/2005, Kyle Pero wrote:
>Unfortunately, the fate of our user tests fall in the hands of recruiting
>companies doing their job correctly. It's not ideal, and I would recruit
>users if I could, but it's the way it is. I think recruiting companies need to
>step up to the plate and do a better job rather than us doing it for them.
As we were preparing our report on recruiting users, we did interviewed
several dozen usability practitioners to understand what their recruiting
challenges and processes are. While the majority of practitioners used
outside recruiting firms, they all complained of severe quality issues with
the participants recruited. They regularly had no-shows, participants that
didn't match the profile, participants that had their own agenda, and
participants that weren't good communicators and therefore didn't help the
team understand their design issues.
The most expensive element of a user study is the time that the participant
is in the chair. If the participant doesn't show up, you waste money. If
they don't give you the feedback about the design that you need, you waste
money. If they give you feedback that doesn't match how 'real users' will
interact with the design, you waste valuable design resources. Ensuring
quality study participants could be one of the most critical elements of a
After doing our research, I came to the conclusion that making the decision
to outsource shouldn't be taken lightly. As Kyle says, it has critical
impact on the quality of our work. If a study fails to meet out client's
need, they aren't going to care if the study was executed perfectly, but
the outside recruiting firm was at fault.
In our research study, we came across several elements that could help
ensure more successful recruiting:
1) The recruiting process is made up of several different components:
Sourcing (the compilation of the study's potential candidate pool),
Contacting (the initial outgoing effort to select potential candidates to
interview), Interviewing (often called Screening, to select participants),
Scheduling (to fit participants into a specific study slot), and
Preparation (to get the participant ready for the session, including
reminder calls). Not all components need to be or should be outsourced.
It's important to make sure each component is covered and you are using the
right resources for each.
2) The best teams had recruiters who knew the products and understood what
made a good study participant. They would conduct open-ended interviews
instead of using a flowchart screener, allowing the recruiter to get a real
"feel" for the participants.
3) The most successful teams met with the recruiter regularly (every couple
of participants scheduled) to discuss what was happening during the
recruitment process and to review the candidates thus far. Even candidates
that aren't qualified for the study yield interesting information about the
definition of the design's potential audience.
4) How is your recruiter rewarded (paid)? Do they get paid the same if they
produce a body that doesn't match your needs as they do if they produce the
best possible participant? How do you ensure they are motivated to put in
the significant extra effort required to get perfect participants for your
study? The best teams had reward structures (bonuses, further work)
explicit in their service agreements to ensure quality of results.
All of these issues are independent of whether your outsourcing components
or doing them within your own organization.
At the risk of being overtly commercial, we talk about this in more detail
in our report, Recruiting Without Fear. You can find more info here:
http://www.uie.com/reports/recruiting_without_fear/ (I know the report is a
little pricey and this is important for CHI Web readers, so I've arranged a
10% discount if you use the CHIWEB promotion code when you order.)
Hope this helps,
Jared M. Spool, Founding Principal
User Interface Engineering
4 Lookout Lane, Unit 4d
Middleton, MA 01949
[log in to unmask]
Tip of the Day: Email mailto:[log in to unmask]
with any comments, questions or problems
CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[log in to unmask]
MODERATORS: mailto:[log in to unmask]
SUBSCRIPTION CHANGES & FAQ: www.sigchi.org/web/faq.html