LISTSERV mailing list manager LISTSERV 16.0

Help for CHI-WEB Archives


CHI-WEB Archives

CHI-WEB Archives


CHI-WEB@LISTSERV.ACM.ORG


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CHI-WEB Home

CHI-WEB Home

CHI-WEB  May 2003, Week 3

CHI-WEB May 2003, Week 3

Subject:

SUMMARY: how many participants?

From:

Pat Malecek <[log in to unmask]>

Reply-To:

Pat Malecek <[log in to unmask]>

Date:

Mon, 19 May 2003 15:12:54 +0000

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (321 lines)

Hi. Thanks to all who responded. It's been recently suggested that posters
of summaries attempt to call out findings/generalizations from the
respondents first and then tack on the verbatim responses. So, here's what I
gleaned (from 7 responses):

- It's impossible to answer unconditionally (as expected), but generally
speaking, all respondents utilized fairly small groups to test (answers
ranged from 3 to 12, even 1 for fatal flaws) and ran multiple sessions. But
to make good use of small groups, the thing(s) being tested and/or the task
set must be constrained. I got a chuckle from the following notion from Bart
Schutz: "5 respondents can really be enough. To convince clients we usually
upgrade this to 8 or 10." Indeed.

- There is general recognition that statistical significance (or even just
impressively large numbers of participants) is not something usability
practitioners can sensibly pursue, nor is it something that clients seem to
be seeking. And that, frankly, is what I was fishing for. I think that helps
us shape how we promote our services. As stated by Gilbert Cockton: "User
testing is a risk reduction exercise. It is not a risk elimination
exercise."

- Three respondents referenced Nielsen's research on the topic. Spool and
Molich were also referenced, as was a thread at usabilitynews.com.

Thanks again. My original post follows, and then you'll find the verbatim
responses.

- Pat

+ + + +

ORIGINAL POST:

UIE recently put out an article regarding 'how many participants is enough?'
(See: http://www.uie.com/Articles/eight_is_not_enough.htm).

Without getting too conditional in your answers, I'd be interested to know
-- generally -- how many participants you use for a typical test (and if
it's 20 participants over 4 test events vs. 10 participants in 1 test event,
please indicate as such).

What I'm fishing for is: In this "real business world" in which many of us
operate, what number of participants has worked for you in freeing you from
folks refuting your findings by saying, "Yeah, well, you only tested with 8
people."

For a bit of context, we involve the product sponsor in the recruiting
process, so s/he is fully aware of the type of individual we're recruiting.
And we don't present our findings as Scientific -- more as supporting
material for the recommendations we make. So, there's no suggestion of
statistical significance.

Please respond to me and I'll post a summary. Thanks! -- Pat

+ + + +

RESPONSES:

In my own work, I've found that Nielsen's recommendations for the numbers of
test participants you need at

http://www.useit.com/alertbox/20000319.html

to be very good, and I frequently conduct tests with 6 to 12 participants.
Important points made by Nielsen that are frequently overlooked are

- If your user population includes highly distinct groups of users, you need
more participants.

- If each test participant can't attempt all of the important tasks that a
product is designed to support in a single test session, you need either
more participants or more or longer test sessions with the same
participants.

- Iterative testing is the most efficient way to test. The goal is to find
problems; bring in too many participants and you waste your time seeing the
same problems again and again. Test quickly, address the problems you find,
and test again.

You can't test Yahoo! with five participants, and Nielsen's recommendations
make that clear.

Mitchell Gass
uLab | PDA: Learning from Users | Designing with Users
Berkeley, CA 94707 USA
+1 510 525-6864 voice
+1 510 525-4246 fax
http://www.participatorydesign.com/

+ + + +

Hi Pat,

Unconditional answer: I tend to plan 2 test events with 4 to 10 participants
each for every release.

Conditional answer: Here I have to agree with what Rolf Molich said during
the CHI2003 panel about this topic. Depending on the *goal* of the test, you
can vary the numbers:

Goal            | #particpants
--------------------------------
sell usablity                   | 3-4
drive development       | 5-8
find all problems       | >100

And this would also take into account the conditions that the original
Nielsen/Landauer article set for the tests:
- test early and test often
- use discount usability methods
- testing is part of a larger UCD process
- expect qualitative data, not quantitative

Peter
--
Peter Boersma, Senior Information Architect, EzGov
"De Schinkel", Rijnsburgstraat 11, Amsterdam, 1059 AT, The Netherlands
Phone: +31(0)20 7133881 / Fax: +31(0)20 7133799 / Mobile: +31(0)6 15072747
mailto:[log in to unmask] / http://www.europe.ezgov.com

+ + + +

See the CHI panel reports for a start at http://www.usabilitynews.com

The fact is one *has* to be conditional in one's answers - anything else is
self-deception.

Good user testing is planned in collaboration with the client, who
understands and accepts the risks inherent in the process, many of which are
not related to the number of test users.  The client has to
arrive at a position where the risks of not testing are far worse than the
risks of testing as planned.  User testing is a risk reduction exercise.  It
is not a risk elimination exercise.  What you do depends on which risks are
most critical and must be checked for and eliminated to some degree of
certainty.

BTW, this is the real world talking as well - none of the top practitioners
start with a number for user testing.  They plan the tests and work out an
initial number which can and does get increased to
better manage specific risks - or it can get drastically reduced, sometimes
to 1 if a fatal usability flaw shows up in the first test.

Good luck
Gilbert

------------------------------------------------------------------------------
Prof. Gilbert Cockton FBCS FRSA,
Research Chair in Human-Computer Interaction, School of Computing &
Technology
                     University of Sunderland, PO Box 299, Sunderland, SR6
0YN
Courier: David Goldman Informatics Centre, St. Peter's Way, Sunderland, SR6
0DD
Tel/Fax: +44 191 515 3394/2781   Web: www.cet.sunderland.ac.uk/~cs0gco
British HCI Group Chair - www.bcs-hci.org.uk - sponsor of
www.usabilitynews.com

+ + + +

Pat,

I have found 3-5 participants in two sessions is sufficient. Usually an
average of four is fine, and having two sessions can help iron out anything
that is deopendant on the circumstances of a session. Ecen with as few as 6
people over the two sessions I have found sufficient variety to suffice for
my needs.

Usually I would go for one session in the morning and another in the
afternoon of the same day, with half the observers changing, so you keep
some involved with both sessions and others who only did one. This brings
out 'afternoon bias', but also gives you some comparative observations.
Obviously a lot depends on the skill of your observers.

Finally, I suppose this too is obvious, but of course the key dependancy is
what you want to get out of your sessions. The sessions I have described
tend to bring out issues of un-usability, and help identify those design
assumptions that were incorrect. What they don't do is demonstrate a
usability quotient, as
it were.

The studies are therefore more adept at identifying negatives than
amplifying/measuring positives. Perhaps this is because we use a kind of
web-ethnography, as it were, managing sessions informally in naturalistic
settings and with high interaction between participants and observers. With
any
ethnographic ("new paradigm") approach, generalisation of results tends to
be dangerous, but these arer well-known methodological constraints. On the
plus side the feedback is far richer than a lab approach, and offer deeper
insights into how people use the web and what presuppositions they have.

You can still use this approach to compare like-for-like sites (Insurance
quote engines for example), but you cannot determine on a single site test a
"score" for how positively usable that site is. I would challenge any other
methodology for doing that, though, other than perhaps just watching
real-world use through sales :)

Hope this is useful,

David Unsworth
Head of Consultancy
Saga Services Ltd
Folkestone, Kent, UK

+ + + +

Pat,

  I typically recommend 5-8 (i.e., shoot for around 8, allowing a bit of
room for a dropout or two... make sure I get 5 or 6).  Like you, I don't
present the results as scientific, merely another diagnostic tool to help
validate things and gather information for future consideration.

jeff

---
Jeff Horvath, Ph.D.
User Centered Design Lead
Berbee
5520 Research Park Dr., Madison, WI 53711
[log in to unmask]
(608) 298-1249;  fax: (608) 288-3007

+ + + +

Pat,

I've been performing usability tests since 1999 (day-in-day-out).  Most
studies I did/ do used 16 to 24 users. Consisting of 2 ('iterative') tests
using 8 or 12 (2x6) single-interviews/ respondents each.

However, this does not imply that there exists an 'ideal number of
respondents'. It depends mostly on targetgroup and size and quality of the
website.

When testing a small (part of a) site that caters for a 'homogeneous'
targetgroup (let's say an online CD-shop/ order process, with a
user-centered design), 5 respondents can really be enough. To convince
clients we usually upgrade this to 8 or 10 (the final interviews usually
unveil only minor findings or are used to test new/ extra ideas).

When testing bigger websites, badly build and catering for different
targetgroups the number increases quickly. For example: I once tested
kpn.com (the Dutch telecom operator), the site had about 6 'main channels',
is very user-unfriendly and kpn distinguishes between business and consumer
users. By
having the respondents perform tasks on mutiple channels, we tried to reduce
the number (and costs). We ended up using 36 respondents (2x6x(6/2)). After
we finished, I had to draw the conclusion it hadn't been enough.

Regards,
Bart Schutz

----------------------------
drs. Bart Schutz
Senior Research Consultant
Manager Qualitative Research

MetrixLab Netherlands
Eendrachtsweg 36
3012 LC Rotterdam
The Netherlands

www.metrixlab.com
[log in to unmask]

tel: +31 (0)10 282 74 68
fax: +31 (0)10 282 74 69
----------------------------

+ + + +

Ahhh, the perennial sticky question...

Here's my take on the whole thing, and I think it jives pretty well with
both Jarod's and Jakob's perspective.

The answer is: it all depends.

I usually feel okay about using a handful of participants if the test
involves a very specific task.  For application testing, I find one, two, or
at most three tasks that I can have participants drive through over the
course of an hour.  In these types of sessions, the comments, feedback and
areas of confusion/delight begin to show strong overlap after three subjects
and very strong tendencies by five subjects.  The key here of course is to
create a very limited and constrained task path.  In these types of tests,
errors stand out in stark relief.

For broader investigations, I avoid usability tests altogether because of
the expense.  In these cases, I up my participant count to as few as 10 and
as many as 60.  For the low end of the spectrum I will engage in one-on-one
interviews, and capture as many as possible, but hopefully 10.  Or I might
use paper prototyping sessions with as many as 10 respondents.  Or if
possible, participatory design sessions.  These moderately low numbers are
pretty effective at getting the kinds of broader questions highlighted and
maybe even a definitive answer or two.

For the higher end of the spectrum I go to more statistically appropriate
techniques, like card sorting.  Naturally, the types of questions I can
answer become much broader, but the answers can be quite defined.

HTH,
  Leo Frishberg
  Senior Information Architect

  W H I T E   H O R S E
  4640 SW Macadam Ave. Suite 100.
  Portland, Oregon 97201.
  503.471.4200 / Fax: 503.471.4299.
  http://www.whitehorse.com.

_________________________________________________________________
Help STOP SPAM with the new MSN 8 and get 2 months FREE*
http://join.msn.com/?page=features/junkmail

    --------------------------------------------------------------
       Tip of the Day: Don't use automated replies to postings
     CHI-WEB: www.sigchi.org/web POSTINGS: mailto:[log in to unmask]
              MODERATORS: mailto:[log in to unmask]
       SUBSCRIPTION CHANGES & FAQ:  www.sigchi.org/web/faq.html
    --------------------------------------------------------------

Top of Message | Previous Page | Permalink

Advanced Options


Options

Log In

Log In

Get Password

Get Password


Search Archives

Search Archives


Subscribe or Unsubscribe

Subscribe or Unsubscribe


Archives

July 2017, Week 2
June 2017, Week 3
May 2017, Week 1
March 2017, Week 5
March 2017, Week 1
January 2017, Week 4
November 2016, Week 1
September 2016, Week 3
September 2016, Week 1
August 2016, Week 5
August 2016, Week 1
July 2016, Week 2
June 2016, Week 1
April 2016, Week 3
April 2016, Week 2
April 2016, Week 1
March 2016, Week 3
February 2016, Week 4
January 2016, Week 4
January 2016, Week 3
September 2015, Week 3
August 2015, Week 5
August 2015, Week 2
July 2015, Week 3
June 2015, Week 4
June 2015, Week 3
June 2015, Week 1
May 2015, Week 2
April 2015, Week 2
April 2015, Week 1
March 2015, Week 5
March 2015, Week 1
February 2015, Week 4
February 2015, Week 2
January 2015, Week 2
December 2014, Week 1
November 2014, Week 4
November 2014, Week 2
October 2014, Week 4
September 2014, Week 5
July 2014, Week 3
July 2014, Week 2
July 2014, Week 1
April 2014, Week 5
April 2014, Week 4
April 2014, Week 3
April 2014, Week 1
March 2014, Week 4
March 2014, Week 3
March 2014, Week 2
February 2014, Week 2
January 2014, Week 4
January 2014, Week 3
January 2014, Week 2
December 2013, Week 4
December 2013, Week 3
November 2013, Week 2
November 2013, Week 1
October 2013, Week 5
October 2013, Week 2
September 2013, Week 5
September 2013, Week 1
August 2013, Week 3
August 2013, Week 2
July 2013, Week 4
July 2013, Week 1
May 2013, Week 2
April 2013, Week 3
April 2013, Week 1
March 2013, Week 3
February 2013, Week 4
February 2013, Week 3
February 2013, Week 1
January 2013, Week 4
January 2013, Week 1
December 2012, Week 4
December 2012, Week 3
December 2012, Week 1
November 2012, Week 4
November 2012, Week 3
November 2012, Week 2
November 2012, Week 1
October 2012, Week 3
September 2012, Week 3
July 2012, Week 4
July 2012, Week 1
June 2012, Week 4
June 2012, Week 3
June 2012, Week 1
May 2012, Week 3
May 2012, Week 2
April 2012, Week 2
April 2012, Week 1
March 2012, Week 1
February 2012, Week 5
February 2012, Week 4
January 2012, Week 5
January 2012, Week 3
January 2012, Week 2
December 2011, Week 3
December 2011, Week 2
December 2011, Week 1
November 2011, Week 2
November 2011, Week 1
October 2011, Week 3
October 2011, Week 1
September 2011, Week 4
September 2011, Week 2
June 2011, Week 1
May 2011, Week 5
May 2011, Week 4
May 2011, Week 2
May 2011, Week 1
April 2011, Week 5
April 2011, Week 4
March 2011, Week 5
March 2011, Week 3
March 2011, Week 2
February 2011, Week 3
February 2011, Week 2
December 2010, Week 5
November 2010, Week 2
October 2010, Week 4
October 2010, Week 2
September 2010, Week 5
September 2010, Week 4
August 2010, Week 4
August 2010, Week 3
August 2010, Week 1
July 2010, Week 5
July 2010, Week 4
July 2010, Week 3
June 2010, Week 5
June 2010, Week 3
June 2010, Week 2
May 2010, Week 4
April 2010, Week 2
March 2010, Week 1
February 2010, Week 3
January 2010, Week 5
January 2010, Week 3
January 2010, Week 2
December 2009, Week 5
December 2009, Week 4
November 2009, Week 4
November 2009, Week 3
November 2009, Week 2
November 2009, Week 1
October 2009, Week 4
October 2009, Week 3
October 2009, Week 1
September 2009, Week 4
September 2009, Week 3
September 2009, Week 1
August 2009, Week 5
August 2009, Week 4
August 2009, Week 3
August 2009, Week 2
August 2009, Week 1
July 2009, Week 5
July 2009, Week 4
July 2009, Week 3
July 2009, Week 2
July 2009, Week 1
June 2009, Week 5
June 2009, Week 4
June 2009, Week 3
June 2009, Week 2
June 2009, Week 1
May 2009, Week 5
May 2009, Week 4
May 2009, Week 3
May 2009, Week 2
April 2009, Week 4
April 2009, Week 3
April 2009, Week 2
April 2009, Week 1
March 2009, Week 4
March 2009, Week 3
March 2009, Week 2
March 2009, Week 1
February 2009, Week 4
January 2009, Week 4
January 2009, Week 2
December 2008, Week 4
December 2008, Week 3
December 2008, Week 2
December 2008, Week 1
November 2008, Week 3
November 2008, Week 2
November 2008, Week 1
October 2008, Week 4
October 2008, Week 3
October 2008, Week 1
September 2008, Week 5
September 2008, Week 4
September 2008, Week 3
September 2008, Week 2
September 2008, Week 1
August 2008, Week 4
August 2008, Week 3
August 2008, Week 2
August 2008, Week 1
July 2008, Week 5
July 2008, Week 4
July 2008, Week 3
June 2008, Week 4
June 2008, Week 1
May 2008, Week 2
May 2008, Week 1
April 2008, Week 5
April 2008, Week 4
April 2008, Week 3
April 2008, Week 2
April 2008, Week 1
March 2008, Week 5
March 2008, Week 1
February 2008, Week 4
February 2008, Week 3
January 2008, Week 5
January 2008, Week 2
January 2008, Week 1
December 2007, Week 3
December 2007, Week 2
December 2007, Week 1
November 2007, Week 5
November 2007, Week 4
November 2007, Week 3
November 2007, Week 2
November 2007, Week 1
October 2007, Week 5
October 2007, Week 4
October 2007, Week 3
October 2007, Week 1
September 2007, Week 5
September 2007, Week 4
September 2007, Week 2
September 2007, Week 1
August 2007, Week 3
August 2007, Week 2
August 2007, Week 1
July 2007, Week 4
July 2007, Week 3
July 2007, Week 2
July 2007, Week 1
June 2007, Week 4
June 2007, Week 3
June 2007, Week 2
June 2007, Week 1
May 2007, Week 5
May 2007, Week 4
May 2007, Week 3
April 2007, Week 4
April 2007, Week 3
April 2007, Week 2
March 2007, Week 3
March 2007, Week 2
March 2007, Week 1
February 2007, Week 4
February 2007, Week 3
February 2007, Week 2
February 2007, Week 1
January 2007, Week 5
January 2007, Week 4
January 2007, Week 3
December 2006, Week 4
December 2006, Week 3
December 2006, Week 2
December 2006, Week 1
November 2006, Week 5
November 2006, Week 4
November 2006, Week 3
November 2006, Week 2
November 2006, Week 1
October 2006, Week 5
October 2006, Week 4
October 2006, Week 3
October 2006, Week 2
October 2006, Week 1
September 2006, Week 5
September 2006, Week 4
September 2006, Week 3
September 2006, Week 2
September 2006, Week 1
August 2006, Week 5
August 2006, Week 4
August 2006, Week 3
August 2006, Week 2
August 2006, Week 1
July 2006, Week 4
July 2006, Week 3
July 2006, Week 2
July 2006, Week 1
June 2006, Week 5
June 2006, Week 4
June 2006, Week 3
June 2006, Week 2
June 2006, Week 1
May 2006, Week 5
May 2006, Week 4
May 2006, Week 3
May 2006, Week 2
May 2006, Week 1
April 2006, Week 5
April 2006, Week 4
April 2006, Week 3
April 2006, Week 2
April 2006, Week 1
March 2006, Week 4
March 2006, Week 3
March 2006, Week 2
March 2006, Week 1
February 2006, Week 4
February 2006, Week 3
February 2006, Week 2
February 2006, Week 1
January 2006, Week 5
January 2006, Week 4
January 2006, Week 3
January 2006, Week 2
January 2006, Week 1
December 2005, Week 4
December 2005, Week 3
December 2005, Week 2
December 2005, Week 1
November 2005, Week 5
November 2005, Week 4
November 2005, Week 3
November 2005, Week 2
November 2005, Week 1
October 2005, Week 5
October 2005, Week 4
October 2005, Week 3
October 2005, Week 2
October 2005, Week 1
September 2005, Week 5
September 2005, Week 4
September 2005, Week 3
September 2005, Week 2
September 2005, Week 1
August 2005, Week 5
August 2005, Week 4
August 2005, Week 3
August 2005, Week 2
August 2005, Week 1
July 2005, Week 5
July 2005, Week 4
July 2005, Week 3
July 2005, Week 2
July 2005, Week 1
June 2005, Week 4
June 2005, Week 3
June 2005, Week 2
June 2005, Week 1
May 2005, Week 5
May 2005, Week 4
May 2005, Week 3
May 2005, Week 2
May 2005, Week 1
April 2005, Week 5
April 2005, Week 4
April 2005, Week 3
April 2005, Week 2
April 2005, Week 1
March 2005, Week 5
March 2005, Week 4
March 2005, Week 3
March 2005, Week 2
March 2005, Week 1
February 2005, Week 4
February 2005, Week 3
February 2005, Week 2
February 2005, Week 1
January 2005, Week 5
January 2005, Week 4
January 2005, Week 3
January 2005, Week 2
January 2005, Week 1
December 2004, Week 5
December 2004, Week 4
December 2004, Week 3
December 2004, Week 2
December 2004, Week 1
November 2004, Week 5
November 2004, Week 4
November 2004, Week 3
November 2004, Week 2
November 2004, Week 1
October 2004, Week 5
October 2004, Week 4
October 2004, Week 3
October 2004, Week 2
October 2004, Week 1
September 2004, Week 5
September 2004, Week 4
September 2004, Week 3
September 2004, Week 2
September 2004, Week 1
August 2004, Week 5
August 2004, Week 4
August 2004, Week 3
August 2004, Week 2
August 2004, Week 1
July 2004, Week 5
July 2004, Week 4
July 2004, Week 3
July 2004, Week 2
July 2004, Week 1
June 2004, Week 5
June 2004, Week 4
June 2004, Week 3
June 2004, Week 2
June 2004, Week 1
May 2004, Week 5
May 2004, Week 4
May 2004, Week 3
May 2004, Week 2
May 2004, Week 1
April 2004, Week 5
April 2004, Week 4
April 2004, Week 3
April 2004, Week 2
April 2004, Week 1
March 2004, Week 1
February 2004, Week 4
February 2004, Week 2
February 2004, Week 1
January 2004, Week 4
January 2004, Week 3
January 2004, Week 2
January 2004, Week 1
December 2003, Week 4
December 2003, Week 3
December 2003, Week 2
December 2003, Week 1
November 2003, Week 5
November 2003, Week 4
November 2003, Week 3
November 2003, Week 2
November 2003, Week 1
October 2003, Week 5
October 2003, Week 4
October 2003, Week 3
October 2003, Week 2
October 2003, Week 1
September 2003, Week 5
September 2003, Week 4
September 2003, Week 3
September 2003, Week 2
September 2003, Week 1
August 2003, Week 5
August 2003, Week 4
August 2003, Week 3
August 2003, Week 2
August 2003, Week 1
July 2003, Week 5
July 2003, Week 4
July 2003, Week 3
July 2003, Week 2
July 2003, Week 1
June 2003, Week 5
June 2003, Week 4
June 2003, Week 3
June 2003, Week 2
June 2003, Week 1
May 2003, Week 5
May 2003, Week 4
May 2003, Week 3
May 2003, Week 2
May 2003, Week 1
April 2003, Week 5
April 2003, Week 4
April 2003, Week 3
April 2003, Week 2
April 2003, Week 1
March 2003, Week 5
March 2003, Week 4
March 2003, Week 3
March 2003, Week 2
March 2003, Week 1
February 2003, Week 4
February 2003, Week 3
February 2003, Week 2
February 2003, Week 1
January 2003, Week 5
January 2003, Week 4
January 2003, Week 3
January 2003, Week 2
January 2003, Week 1
December 2002, Week 5
December 2002, Week 4
December 2002, Week 3
December 2002, Week 2
December 2002, Week 1
November 2002, Week 5
November 2002, Week 4
November 2002, Week 3
November 2002, Week 2
November 2002, Week 1
October 2002, Week 5
October 2002, Week 4
October 2002, Week 3
October 2002, Week 2
October 2002, Week 1
September 2002, Week 5
September 2002, Week 4
September 2002, Week 3
September 2002, Week 2
September 2002, Week 1
August 2002, Week 4
August 2002, Week 3
August 2002, Week 2
August 2002, Week 1
July 2002, Week 5
July 2002, Week 4
July 2002, Week 3
July 2002, Week 2
July 2002, Week 1
June 2002, Week 5
June 2002, Week 4
June 2002, Week 3
June 2002, Week 2
June 2002, Week 1
May 2002, Week 5
May 2002, Week 4
May 2002, Week 3
May 2002, Week 2
May 2002, Week 1
April 2002, Week 5
April 2002, Week 4
April 2002, Week 3
April 2002, Week 2
April 2002, Week 1
March 2002, Week 5
March 2002, Week 4
March 2002, Week 3
March 2002, Week 2
March 2002, Week 1
February 2002, Week 4
February 2002, Week 3
February 2002, Week 2
February 2002, Week 1
January 2002, Week 5
January 2002, Week 4
January 2002, Week 3
January 2002, Week 2
January 2002, Week 1
December 2001, Week 5
December 2001, Week 4
December 2001, Week 3
December 2001, Week 2
December 2001, Week 1
November 2001, Week 5
November 2001, Week 3
November 2001, Week 1
October 2001, Week 5
October 2001, Week 4
October 2001, Week 3
October 2001, Week 2
October 2001, Week 1
September 2001, Week 5
September 2001, Week 4
September 2001, Week 3
September 2001, Week 2
September 2001, Week 1
August 2001, Week 5
August 2001, Week 4
August 2001, Week 3
August 2001, Week 2
August 2001, Week 1
July 2001, Week 5
July 2001, Week 4
July 2001, Week 3
July 2001, Week 2
July 2001, Week 1
June 2001, Week 5
June 2001, Week 4
June 2001, Week 3
June 2001, Week 2
June 2001, Week 1
May 2001, Week 5
May 2001, Week 4
May 2001, Week 3
May 2001, Week 2
May 2001, Week 1
April 2001, Week 5
April 2001, Week 4
April 2001, Week 3
April 2001, Week 2
April 2001, Week 1
March 2001, Week 5
March 2001, Week 4
March 2001, Week 3
March 2001, Week 2
March 2001, Week 1
February 2001, Week 4
February 2001, Week 3
February 2001, Week 2
February 2001, Week 1
January 2001, Week 5
January 2001, Week 4
January 2001, Week 3
January 2001, Week 2
January 2001, Week 1
December 2000, Week 5
December 2000, Week 4
December 2000, Week 3
December 2000, Week 2
December 2000, Week 1
November 2000, Week 5
November 2000, Week 4
November 2000, Week 3
November 2000, Week 2
November 2000, Week 1
October 2000, Week 5
October 2000, Week 4
October 2000, Week 3
October 2000, Week 2
October 2000, Week 1
September 2000, Week 5
September 2000, Week 4
September 2000, Week 3
September 2000, Week 2
September 2000, Week 1
August 2000, Week 5
August 2000, Week 4
August 2000, Week 3
August 2000, Week 2
August 2000, Week 1
July 2000, Week 5
July 2000, Week 4
July 2000, Week 3
July 2000, Week 2
July 2000, Week 1
June 2000, Week 5
June 2000, Week 4
June 2000, Week 3
June 2000, Week 2
June 2000, Week 1
May 2000, Week 5
May 2000, Week 4
May 2000, Week 3
May 2000, Week 2
May 2000, Week 1
April 2000, Week 5
April 2000, Week 4
April 2000, Week 3
April 2000, Week 2
April 2000, Week 1
March 2000, Week 5
March 2000, Week 4
March 2000, Week 3
March 2000, Week 2
March 2000, Week 1
February 2000, Week 5
February 2000, Week 4
February 2000, Week 3
February 2000, Week 2
February 2000, Week 1
January 2000, Week 5
January 2000, Week 4
January 2000, Week 3
January 2000, Week 2
January 2000, Week 1
December 1999, Week 5
December 1999, Week 4
December 1999, Week 3
December 1999, Week 2
December 1999, Week 1
November 1999, Week 5
November 1999, Week 4
November 1999, Week 3
November 1999, Week 2
November 1999, Week 1
October 1999, Week 5
October 1999, Week 4
October 1999, Week 3
October 1999, Week 2
October 1999, Week 1
September 1999, Week 5
September 1999, Week 4
September 1999, Week 3
September 1999, Week 2
September 1999, Week 1
August 1999, Week 5
August 1999, Week 4
August 1999, Week 3
August 1999, Week 2
August 1999, Week 1
July 1999, Week 5
July 1999, Week 4
July 1999, Week 3
July 1999, Week 2
July 1999, Week 1
June 1999, Week 5
June 1999, Week 4
June 1999, Week 3
June 1999, Week 2
June 1999, Week 1
May 1999, Week 4
May 1999, Week 3
May 1999, Week 2
May 1999, Week 1
April 1999, Week 5
April 1999, Week 4
April 1999, Week 3
April 1999, Week 2
April 1999, Week 1
March 1999, Week 5
March 1999, Week 4
March 1999, Week 3
March 1999, Week 2
March 1999, Week 1
February 1999, Week 4
February 1999, Week 3
February 1999, Week 2
February 1999, Week 1
January 1999, Week 5
January 1999, Week 4
January 1999, Week 3
January 1999, Week 2
January 1999, Week 1
December 1998, Week 5
December 1998, Week 3
December 1998, Week 2
December 1998, Week 1
November 1998, Week 5
November 1998, Week 4
November 1998, Week 3
November 1998, Week 2
November 1998, Week 1
October 1998, Week 5
October 1998, Week 4
October 1998, Week 3
October 1998, Week 2
October 1998, Week 1
September 1998, Week 4
September 1998, Week 3
September 1998, Week 2
September 1998, Week 1
August 1998, Week 5
August 1998, Week 4
August 1998, Week 3
August 1998, Week 2
August 1998, Week 1
July 1998, Week 5
July 1998, Week 4
July 1998, Week 3
July 1998, Week 2
July 1998, Week 1
June 1998, Week 4
June 1998, Week 3
June 1998, Week 2
June 1998, Week 1
May 1998, Week 4
May 1998, Week 3
May 1998, Week 2
May 1998, Week 1
April 1998, Week 5
April 1998, Week 4
April 1998, Week 3
April 1998, Week 2
April 1998, Week 1
March 1998, Week 5
March 1998, Week 4
March 1998, Week 3
March 1998, Week 2
March 1998, Week 1
February 1998, Week 2
February 1998, Week 1
January 1998, Week 4
January 1998, Week 3
January 1998, Week 2
January 1998, Week 1
December 1997, Week 4
December 1997, Week 3
December 1997, Week 2
December 1997, Week 1
November 1997, Week 4
November 1997, Week 2
October 1997, Week 3
September 1997, Week 5
September 1997, Week 4
September 1997, Week 3
September 1997, Week 1
August 1997, Week 4
August 1997, Week 2
August 1997, Week 1
July 1997, Week 4
July 1997, Week 3
July 1997, Week 2
June 1997, Week 5
June 1997, Week 4
June 1997, Week 2
June 1997, Week 1
May 1997, Week 5
May 1997, Week 4
May 1997, Week 3
April 1997, Week 5
April 1997, Week 4
April 1997, Week 3
April 1997, Week 2
April 1997, Week 1
March 1997, Week 4
March 1997, Week 3
March 1997, Week 2
March 1997, Week 1
February 1997, Week 4
February 1997, Week 3
February 1997, Week 2
February 1997, Week 1
January 1997, Week 5

ATOM RSS1 RSS2



LISTSERV.ACM.ORG

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager