At 09:06 AM 3/29/2005, Kyle Pero wrote:
>Unfortunately, the fate of our user tests fall in the hands of recruiting
>companies doing their job correctly. It's not ideal, and I would recruit 
>my own
>users if I could, but it's the way it is. I think recruiting companies need to
>step up to the plate and do a better job rather than us doing it for them.

As we were preparing our report on recruiting users, we did interviewed 
several dozen usability practitioners to understand what their recruiting 
challenges and processes are. While the majority of practitioners used 
outside recruiting firms, they all complained of severe quality issues with 
the participants recruited. They regularly had no-shows, participants that 
didn't match the profile, participants that had their own agenda, and 
participants that weren't good communicators and therefore didn't help the 
team understand their design issues.

The most expensive element of a user study is the time that the participant 
is in the chair. If the participant doesn't show up, you waste money. If 
they don't give you the feedback about the design that you need, you waste 
money. If they give you feedback that doesn't match how 'real users' will 
interact with the design, you waste valuable design resources. Ensuring 
quality study participants could be one of the most critical elements of a 
successful study.

After doing our research, I came to the conclusion that making the decision 
to outsource shouldn't be taken lightly. As Kyle says, it has critical 
impact on the quality of our work. If a study fails to meet out client's 
need, they aren't going to care if the study was executed perfectly, but 
the outside recruiting firm was at fault.

In our research study, we came across several elements that could help 
ensure more successful recruiting:

1) The recruiting process is made up of several different components: 
Sourcing (the compilation of the study's potential candidate pool), 
Contacting (the initial outgoing effort to select potential candidates to 
interview), Interviewing (often called Screening, to select participants), 
Scheduling (to fit participants into a specific study slot), and 
Preparation (to get the participant ready for the session, including 
reminder calls). Not all components need to be or should be outsourced. 
It's important to make sure each component is covered and you are using the 
right resources for each.

2) The best teams had recruiters who knew the products and understood what 
made a good study participant. They would conduct open-ended interviews 
instead of using a flowchart screener, allowing the recruiter to get a real 
"feel" for the participants.

3) The most successful teams met with the recruiter regularly (every couple 
of participants scheduled) to discuss what was happening during the 
recruitment process and to review the candidates thus far. Even candidates 
that aren't qualified for the study yield interesting information about the 
definition of the design's potential audience.

4) How is your recruiter rewarded (paid)? Do they get paid the same if they 
produce a body that doesn't match your needs as they do if they produce the 
best possible participant? How do you ensure they are motivated to put in 
the significant extra effort required to get perfect participants for your 
study? The best teams had reward structures (bonuses, further work) 
explicit in their service agreements to ensure quality of results.

All of these issues are independent of whether your outsourcing components 
or doing them within your own organization.

At the risk of being overtly commercial, we talk about this in more detail 
in our report, Recruiting Without Fear. You can find more info here: 
http://www.uie.com/reports/recruiting_without_fear/ (I know the report is a 
little pricey and this is important for CHI Web readers, so I've arranged a 
10% discount if you use the CHIWEB promotion code when you order.)

Hope this helps,

Jared


Jared M. Spool, Founding Principal
User Interface Engineering
4 Lookout Lane, Unit 4d
Middleton, MA 01949
978 777-9123
[log in to unmask]
http://www.uie.com 

    --------------------------------------------------------------
        Tip of the Day: Email mailto:[log in to unmask]
               with any comments, questions or problems
     CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[log in to unmask]
              MODERATORS: mailto:[log in to unmask]
       SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
    --------------------------------------------------------------