Amoung other things, Rick said:

> Understandably, there are many people in this profession for whom the
> obvious points are new and remarkable.  This isn't a knock on the quality
> the talent or anyone on this list, but an observation of what I have seen
> over time.  I've developed 5 huge ecommerce sites and have specific
> experience with all the issues that arise, so maybe I'm looking in the
> place or expecting more than these studies are geared to deliver.  It just
> seems that we would all be better served by studies that actually STUDY
> subject and offer well tested theorems on the different aspects of
> experience.

I'm seen a lot of usability folks work (I hesitate to use the term engineer,
since very few of them had engineering backgrounds) in the past few years,
and they are mostly bad. bad ranged from obvious badness such as the
moderator grabbing the mouse from the user, to subtle badness such as the
analyst clearly not reviewing the tapes.

Bulk of badness tended to run along these lines:

bad prep-work: ignoring the key issues the hire-er wanted dealt with, or not
going beyond the key issues the hire-er wanted dealt with

bad moderation: leading questions, feedback that shaped user responses,
encouraging/extracting too much focus-group-style feedback (how does that
red make you feel), interruption of user when they are talking, etc.

bad analysis: not explaining how the results were derived, careless analysis
(in one test recently I saw two users exhibit a behavior, the report said no
one did this item, reviewing the tapes showed that it occurred), too many
precise recommendations (if you are a frustrated designer, become a
designer. Don't recommend "blue may help this area recede." it may please a
client, and you may be right, but the designer will not want to work with
you again.)

or so it has run in my experience. of course, some of these items would be
appropriate in some situations, but overall in a typically usability test
this is bad behavior and tends to lead to bad testing. There are amazing
amounts of bad usability folks out there.

   About CHI-WEB: