LISTSERV mailing list manager LISTSERV 16.5

Help for CHI-WEB Archives


CHI-WEB Archives

CHI-WEB Archives


CHI-WEB@LISTSERV.ACM.ORG


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CHI-WEB Home

CHI-WEB Home

CHI-WEB  January 2002, Week 3

CHI-WEB January 2002, Week 3

Subject:

Summary: Search Relevancy

From:

Kayla Block <[log in to unmask]>

Reply-To:

Kayla Block <[log in to unmask]>

Date:

Sun, 20 Jan 2002 08:13:59 -0800

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (227 lines)

I asked about whether providing users with relevancy information was
actually helpful to users doing queries. My suspicion was that relevancy
from an algorithmic perspective is not the same as from a user perspective
and was therefore not helpful.

As it turns out, part of my question arose from a misunderstanding on my
part. Our current search has a checkbox that says "use weighted order." I
interpreted this to mean that results would be ranked in terms of
relevancy. In fact, it means that if you enter in "foo bar baz", "foo" is
deemed to be more important than "bar" or "baz".


Responses to my question fell into two general categories. Those who said
that having a distinction between highly relevant (90-100%) with a big drop
off to barely relevant (30%) helps some users distinguish which results to
pursue further. The other camp believed that relevancy ratings only confuse
users or don't add any value.

Specific responses are summarized below:
This is another of those 'when worlds collide' issues, where software and
users don't share the same definition of 'relevance'.

I've seen some spectacularly unimpressed people when I've user tested search
results pages with '% relevant' columns. People tend to define relevancy as
'relevant to what I was really looking for'.  Obviously, the database
defines relevancy as 'relevant to the query that you just gave me'.  Given
that people are generally fairly poor as specifying good queries, may not
know exactly what they were after until they see it, etc, there is bound to
be a gap between a persons real need for particular information and how they
actually express it to the database.  So I guess one way to think about it
would be as a communication problem.

Solution?  Either make sure that both parties have the same understanding of
'relevancy' (which will almost certainly involve the person understanding
the ranking algorithm), or don't explicitly present relevancy rankings to
the user.  Which option you choose will depend on the experience and
capabilities of your target users.

regards,

Chris Lowe

==============

I agree with you in some ways: the exact numbers are meaningless to
everyone but the search engine designers.

Relevance scores are usually derived from: the term frequency: the number
of word matches in each document, sometimes adjusted for the length of the
document.  In many cases, the scores are reduced by the inverse document
frequency the number of times the word appears in the collection -- with
the assumption that rare words are more valuable.  This is called
TF/IDF.  Better search engines account for other aspects, such as
semistructured HTML (titles are important), number of incoming links, etc.

Oddly enough, there seems to be a wide variety in the form of the score:
some assign the most relevant document 100% and move down from there,
others compare all documents against some kind of Platonic ideal of
relevance and start wherever the best document fits.  Still others use a
statistical style, with 1.0 being perfect: these ones tend to trail off
into decimal meaninglessness.

Some search engines such as ht://Dig use stars or bullets to show
relevance.  That's easier for searchers to recognize, but I think it's
misleading.  In other contexts, such as restaurant or movie reviews, a
human has made a judgement about quality.  In search engine results, it's
just algorithmic text matching, which is often much less useful.

I like the Inktomi Enterprise (nee Ultraseek) approach: it shows the
relevance score as a filled bar, so you can see that some documents contain
significantly more matches than other documents.  I think that's a way to
convey the lack of precision in these rankings, while still offering a
relative relationship.

Thanks for the good question -- got my brain going this morning!

Avi

==============

As an alternative to relevancy ratings we return search results that are
sorted by product category, and the option to sort search results by the
following sort criterion:

    "Price - Low to High"
    "Price -High to Low "
    Alphabetically (A-Z)
    Alphabetically (Z-A)
    Closest Match (to the query they formulated)

Results are initially returned based on closest match as defined by our
business rules and users can opt to sort after results are returned. I
conducted a TAP test with 10 users in our target audience and the behavior
I observed was investigative in nature. For example, within the context of
a task they would say "I wonder what 'closest match' means. It must mean
that the site gave me the products that best matched the words I searched
for."

I am not sure if you were polling our community for alternatives to
relevancy ratings, but I hope this helps. Please omit my name and company
information when posting a summary to the group. Also - if you would like
further information, please feel free to query me about specific search
design practices as needed.

=============

         In the early days of web search engines -- almost all of which use
relevancy algorithms as their major ranking function -- the results weren't
all that great.  The relevancy algorithms have improved greatly since then,
and most search engines allow a mixture of relevancy and boolean searching,
along with other factors such as link popularity, to get rankings.  Google
is the premier example of a search engine whose relevancy algorithms are
superb. (Of course it depends on what your search is.  Google still isn't
very good for precision searching due to its lack of truncation, precise
proximity and other features.)

>But I'm curious whether users find relevancy ratings helpful? If the most
>relevant results are returned to the top anyway, how do the ratings help?
>In general, a search hit is either relevant or not.....rather than
>
>I also don't understand how they are derived.
         Relevancy ratings are a good way to see how much of your search
statement is included in the result.  For instance, you type in a search for:
Britney Spears loves Justin Timberlake  (Britney Spears being one of the
most popular web searches).

100% rating would bring up documents that have the phrase "Britney Spears
loves Justin Timberlake" and that probably have the phrase many times in
the document.
99% rating would bring up documents that almost, but don't exactly match
the search, such as "Britney Spears loves Justin" where the word Timberlake
is somewhere near the phrase.
90% rating might be "Britney loves Justin" where their last names are close
by in the document.
80% rating might have the words "Britney" "Spears" "loves" "Justin" and
"Timberlake" somewhere in the document, but not as close together.
50% rating might mean that one or more of the five words in the search
phrase is not present in the document.
30% rating might mean that only one or two of the words in the search
phrase are in the document.

And so on.  With relevancy ranking you can get results back that have only
one of the five words in the search phrase.  The 10% rating would tell you
that it isn't very likely that that document is going to be relevant to
your purpose.

The advantage of relevancy ranking is that people can type in words and get
relevant documents that don't exactly match the search phrase, but are
still fit their purpose.  For instance, "Justin Timberlake loves Britney
Spears" would be relevant, but wouldn't have 100% rating.  A document that
has a headline or lead paragraph saying "Justin loves Britney" would also
be relevant to most searchers, but would have a lower relevancy rating
because not all words are matched.  Doing Boolean searching you might lose
the latter document is the default was "AND" and the words Timberlake and
Spears weren't in the document.

I hope this helps.

Terry

Terry Brainerd Chadwick, InfoQuest! Information Services
mailto:[log in to unmask]         http://www.tbchad.com
Strategic Research, Competitive Intelligence & Content Development
Website Optimization for Accessibility, Content, Search & Usability

==============

In some testing that I was involved in about 3 years ago on various search
engines we found that people do not find relevance indicators at all useful
and in fact could be counter productive as they confused people when the
result didn't seem particularly relevant to what they wanted. Overall these
tests showed that people believed the search results would show the most
relevant result first and that any kind of counter was redundant.

Hope this is useful
Meaghan

Meaghan Waters | Information Architect | Platinion Pty Ltd
A Company of The Boston Consulting Group
Level 3, 241 Commonwealth Street, Surry Hills, NSW 2010
direct line: 02 8204 7731
fax: 02 8204 7777
m: 0402 029 216
e-mail: [log in to unmask]
www.platinion.com

===========

In our research, we have yet to find a single instance where the display of
relevancy ratings contributed to a user's success in any way.  In fact,
we've observed numerous instances where users were confused by these
ratings and even joked about the relevancy of the relevancy
ratings!  Sometimes the piece of content the user is seeking is rated as
less relevant than other content they could care less about.  So the value
of these ratings is a bit mystifying. Users think everything that search
returns should be "relevant".

In our upcoming Search report, "In Search of the Perfect Search", we
discuss relevancy ratings and other attributes of search that did not help
users complete their tasks - such as sorting / filters. (the report should
be out within the next month and will be announced on our home page and by
email if you are on our list: http://www.uie.com)

Erik Ojakaar
User Interface Engineering
242 Neck Rd
Bradford, MA 01835
http://www.uie.com
978-374-8300

===================

I find them of some, but limited, utlity. If the first six entries are 99%
relevant, for example, and then there's a drop-off to 60% or 40%, I know
not to spend much time with those entries. I do scan the list to find stuff
that's over 75% percent relevant.

Debra Goldentyer

    --------------------------------------------------------------
        Tip of the Day: Forward out-of-office replies to
                    mailto:[log in to unmask]
               About CHI-WEB: http://www.sigchi.org/web/
         Vacations, holidays and other subscription changes:
             http://www.sigchi.org/web/faq.html#vacations
    --------------------------------------------------------------

Top of Message | Previous Page | Permalink

Advanced Options


Options

Log In

Log In

Get Password

Get Password


Search Archives

Search Archives


Subscribe or Unsubscribe

Subscribe or Unsubscribe


Archives

July 2017, Week 2
June 2017, Week 3
May 2017, Week 1
March 2017, Week 5
March 2017, Week 1
January 2017, Week 4
November 2016, Week 1
September 2016, Week 3
September 2016, Week 1
August 2016, Week 5
August 2016, Week 1
July 2016, Week 2
June 2016, Week 1
April 2016, Week 3
April 2016, Week 2
April 2016, Week 1
March 2016, Week 3
February 2016, Week 4
January 2016, Week 4
January 2016, Week 3
September 2015, Week 3
August 2015, Week 5
August 2015, Week 2
July 2015, Week 3
June 2015, Week 4
June 2015, Week 3
June 2015, Week 1
May 2015, Week 2
April 2015, Week 2
April 2015, Week 1
March 2015, Week 5
March 2015, Week 1
February 2015, Week 4
February 2015, Week 2
January 2015, Week 2
December 2014, Week 1
November 2014, Week 4
November 2014, Week 2
October 2014, Week 4
September 2014, Week 5
July 2014, Week 3
July 2014, Week 2
July 2014, Week 1
April 2014, Week 5
April 2014, Week 4
April 2014, Week 3
April 2014, Week 1
March 2014, Week 4
March 2014, Week 3
March 2014, Week 2
February 2014, Week 2
January 2014, Week 4
January 2014, Week 3
January 2014, Week 2
December 2013, Week 4
December 2013, Week 3
November 2013, Week 2
November 2013, Week 1
October 2013, Week 5
October 2013, Week 2
September 2013, Week 5
September 2013, Week 1
August 2013, Week 3
August 2013, Week 2
July 2013, Week 4
July 2013, Week 1
May 2013, Week 2
April 2013, Week 3
April 2013, Week 1
March 2013, Week 3
February 2013, Week 4
February 2013, Week 3
February 2013, Week 1
January 2013, Week 4
January 2013, Week 1
December 2012, Week 4
December 2012, Week 3
December 2012, Week 1
November 2012, Week 4
November 2012, Week 3
November 2012, Week 2
November 2012, Week 1
October 2012, Week 3
September 2012, Week 3
July 2012, Week 4
July 2012, Week 1
June 2012, Week 4
June 2012, Week 3
June 2012, Week 1
May 2012, Week 3
May 2012, Week 2
April 2012, Week 2
April 2012, Week 1
March 2012, Week 1
February 2012, Week 5
February 2012, Week 4
January 2012, Week 5
January 2012, Week 3
January 2012, Week 2
December 2011, Week 3
December 2011, Week 2
December 2011, Week 1
November 2011, Week 2
November 2011, Week 1
October 2011, Week 3
October 2011, Week 1
September 2011, Week 4
September 2011, Week 2
June 2011, Week 1
May 2011, Week 5
May 2011, Week 4
May 2011, Week 2
May 2011, Week 1
April 2011, Week 5
April 2011, Week 4
March 2011, Week 5
March 2011, Week 3
March 2011, Week 2
February 2011, Week 3
February 2011, Week 2
December 2010, Week 5
November 2010, Week 2
October 2010, Week 4
October 2010, Week 2
September 2010, Week 5
September 2010, Week 4
August 2010, Week 4
August 2010, Week 3
August 2010, Week 1
July 2010, Week 5
July 2010, Week 4
July 2010, Week 3
June 2010, Week 5
June 2010, Week 3
June 2010, Week 2
May 2010, Week 4
April 2010, Week 2
March 2010, Week 1
February 2010, Week 3
January 2010, Week 5
January 2010, Week 3
January 2010, Week 2
December 2009, Week 5
December 2009, Week 4
November 2009, Week 4
November 2009, Week 3
November 2009, Week 2
November 2009, Week 1
October 2009, Week 4
October 2009, Week 3
October 2009, Week 1
September 2009, Week 4
September 2009, Week 3
September 2009, Week 1
August 2009, Week 5
August 2009, Week 4
August 2009, Week 3
August 2009, Week 2
August 2009, Week 1
July 2009, Week 5
July 2009, Week 4
July 2009, Week 3
July 2009, Week 2
July 2009, Week 1
June 2009, Week 5
June 2009, Week 4
June 2009, Week 3
June 2009, Week 2
June 2009, Week 1
May 2009, Week 5
May 2009, Week 4
May 2009, Week 3
May 2009, Week 2
April 2009, Week 4
April 2009, Week 3
April 2009, Week 2
April 2009, Week 1
March 2009, Week 4
March 2009, Week 3
March 2009, Week 2
March 2009, Week 1
February 2009, Week 4
January 2009, Week 4
January 2009, Week 2
December 2008, Week 4
December 2008, Week 3
December 2008, Week 2
December 2008, Week 1
November 2008, Week 3
November 2008, Week 2
November 2008, Week 1
October 2008, Week 4
October 2008, Week 3
October 2008, Week 1
September 2008, Week 5
September 2008, Week 4
September 2008, Week 3
September 2008, Week 2
September 2008, Week 1
August 2008, Week 4
August 2008, Week 3
August 2008, Week 2
August 2008, Week 1
July 2008, Week 5
July 2008, Week 4
July 2008, Week 3
June 2008, Week 4
June 2008, Week 1
May 2008, Week 2
May 2008, Week 1
April 2008, Week 5
April 2008, Week 4
April 2008, Week 3
April 2008, Week 2
April 2008, Week 1
March 2008, Week 5
March 2008, Week 1
February 2008, Week 4
February 2008, Week 3
January 2008, Week 5
January 2008, Week 2
January 2008, Week 1
December 2007, Week 3
December 2007, Week 2
December 2007, Week 1
November 2007, Week 5
November 2007, Week 4
November 2007, Week 3
November 2007, Week 2
November 2007, Week 1
October 2007, Week 5
October 2007, Week 4
October 2007, Week 3
October 2007, Week 1
September 2007, Week 5
September 2007, Week 4
September 2007, Week 2
September 2007, Week 1
August 2007, Week 3
August 2007, Week 2
August 2007, Week 1
July 2007, Week 4
July 2007, Week 3
July 2007, Week 2
July 2007, Week 1
June 2007, Week 4
June 2007, Week 3
June 2007, Week 2
June 2007, Week 1
May 2007, Week 5
May 2007, Week 4
May 2007, Week 3
April 2007, Week 4
April 2007, Week 3
April 2007, Week 2
March 2007, Week 3
March 2007, Week 2
March 2007, Week 1
February 2007, Week 4
February 2007, Week 3
February 2007, Week 2
February 2007, Week 1
January 2007, Week 5
January 2007, Week 4
January 2007, Week 3
December 2006, Week 4
December 2006, Week 3
December 2006, Week 2
December 2006, Week 1
November 2006, Week 5
November 2006, Week 4
November 2006, Week 3
November 2006, Week 2
November 2006, Week 1
October 2006, Week 5
October 2006, Week 4
October 2006, Week 3
October 2006, Week 2
October 2006, Week 1
September 2006, Week 5
September 2006, Week 4
September 2006, Week 3
September 2006, Week 2
September 2006, Week 1
August 2006, Week 5
August 2006, Week 4
August 2006, Week 3
August 2006, Week 2
August 2006, Week 1
July 2006, Week 4
July 2006, Week 3
July 2006, Week 2
July 2006, Week 1
June 2006, Week 5
June 2006, Week 4
June 2006, Week 3
June 2006, Week 2
June 2006, Week 1
May 2006, Week 5
May 2006, Week 4
May 2006, Week 3
May 2006, Week 2
May 2006, Week 1
April 2006, Week 5
April 2006, Week 4
April 2006, Week 3
April 2006, Week 2
April 2006, Week 1
March 2006, Week 4
March 2006, Week 3
March 2006, Week 2
March 2006, Week 1
February 2006, Week 4
February 2006, Week 3
February 2006, Week 2
February 2006, Week 1
January 2006, Week 5
January 2006, Week 4
January 2006, Week 3
January 2006, Week 2
January 2006, Week 1
December 2005, Week 4
December 2005, Week 3
December 2005, Week 2
December 2005, Week 1
November 2005, Week 5
November 2005, Week 4
November 2005, Week 3
November 2005, Week 2
November 2005, Week 1
October 2005, Week 5
October 2005, Week 4
October 2005, Week 3
October 2005, Week 2
October 2005, Week 1
September 2005, Week 5
September 2005, Week 4
September 2005, Week 3
September 2005, Week 2
September 2005, Week 1
August 2005, Week 5
August 2005, Week 4
August 2005, Week 3
August 2005, Week 2
August 2005, Week 1
July 2005, Week 5
July 2005, Week 4
July 2005, Week 3
July 2005, Week 2
July 2005, Week 1
June 2005, Week 4
June 2005, Week 3
June 2005, Week 2
June 2005, Week 1
May 2005, Week 5
May 2005, Week 4
May 2005, Week 3
May 2005, Week 2
May 2005, Week 1
April 2005, Week 5
April 2005, Week 4
April 2005, Week 3
April 2005, Week 2
April 2005, Week 1
March 2005, Week 5
March 2005, Week 4
March 2005, Week 3
March 2005, Week 2
March 2005, Week 1
February 2005, Week 4
February 2005, Week 3
February 2005, Week 2
February 2005, Week 1
January 2005, Week 5
January 2005, Week 4
January 2005, Week 3
January 2005, Week 2
January 2005, Week 1
December 2004, Week 5
December 2004, Week 4
December 2004, Week 3
December 2004, Week 2
December 2004, Week 1
November 2004, Week 5
November 2004, Week 4
November 2004, Week 3
November 2004, Week 2
November 2004, Week 1
October 2004, Week 5
October 2004, Week 4
October 2004, Week 3
October 2004, Week 2
October 2004, Week 1
September 2004, Week 5
September 2004, Week 4
September 2004, Week 3
September 2004, Week 2
September 2004, Week 1
August 2004, Week 5
August 2004, Week 4
August 2004, Week 3
August 2004, Week 2
August 2004, Week 1
July 2004, Week 5
July 2004, Week 4
July 2004, Week 3
July 2004, Week 2
July 2004, Week 1
June 2004, Week 5
June 2004, Week 4
June 2004, Week 3
June 2004, Week 2
June 2004, Week 1
May 2004, Week 5
May 2004, Week 4
May 2004, Week 3
May 2004, Week 2
May 2004, Week 1
April 2004, Week 5
April 2004, Week 4
April 2004, Week 3
April 2004, Week 2
April 2004, Week 1
March 2004, Week 1
February 2004, Week 4
February 2004, Week 2
February 2004, Week 1
January 2004, Week 4
January 2004, Week 3
January 2004, Week 2
January 2004, Week 1
December 2003, Week 4
December 2003, Week 3
December 2003, Week 2
December 2003, Week 1
November 2003, Week 5
November 2003, Week 4
November 2003, Week 3
November 2003, Week 2
November 2003, Week 1
October 2003, Week 5
October 2003, Week 4
October 2003, Week 3
October 2003, Week 2
October 2003, Week 1
September 2003, Week 5
September 2003, Week 4
September 2003, Week 3
September 2003, Week 2
September 2003, Week 1
August 2003, Week 5
August 2003, Week 4
August 2003, Week 3
August 2003, Week 2
August 2003, Week 1
July 2003, Week 5
July 2003, Week 4
July 2003, Week 3
July 2003, Week 2
July 2003, Week 1
June 2003, Week 5
June 2003, Week 4
June 2003, Week 3
June 2003, Week 2
June 2003, Week 1
May 2003, Week 5
May 2003, Week 4
May 2003, Week 3
May 2003, Week 2
May 2003, Week 1
April 2003, Week 5
April 2003, Week 4
April 2003, Week 3
April 2003, Week 2
April 2003, Week 1
March 2003, Week 5
March 2003, Week 4
March 2003, Week 3
March 2003, Week 2
March 2003, Week 1
February 2003, Week 4
February 2003, Week 3
February 2003, Week 2
February 2003, Week 1
January 2003, Week 5
January 2003, Week 4
January 2003, Week 3
January 2003, Week 2
January 2003, Week 1
December 2002, Week 5
December 2002, Week 4
December 2002, Week 3
December 2002, Week 2
December 2002, Week 1
November 2002, Week 5
November 2002, Week 4
November 2002, Week 3
November 2002, Week 2
November 2002, Week 1
October 2002, Week 5
October 2002, Week 4
October 2002, Week 3
October 2002, Week 2
October 2002, Week 1
September 2002, Week 5
September 2002, Week 4
September 2002, Week 3
September 2002, Week 2
September 2002, Week 1
August 2002, Week 4
August 2002, Week 3
August 2002, Week 2
August 2002, Week 1
July 2002, Week 5
July 2002, Week 4
July 2002, Week 3
July 2002, Week 2
July 2002, Week 1
June 2002, Week 5
June 2002, Week 4
June 2002, Week 3
June 2002, Week 2
June 2002, Week 1
May 2002, Week 5
May 2002, Week 4
May 2002, Week 3
May 2002, Week 2
May 2002, Week 1
April 2002, Week 5
April 2002, Week 4
April 2002, Week 3
April 2002, Week 2
April 2002, Week 1
March 2002, Week 5
March 2002, Week 4
March 2002, Week 3
March 2002, Week 2
March 2002, Week 1
February 2002, Week 4
February 2002, Week 3
February 2002, Week 2
February 2002, Week 1
January 2002, Week 5
January 2002, Week 4
January 2002, Week 3
January 2002, Week 2
January 2002, Week 1
December 2001, Week 5
December 2001, Week 4
December 2001, Week 3
December 2001, Week 2
December 2001, Week 1
November 2001, Week 5
November 2001, Week 3
November 2001, Week 1
October 2001, Week 5
October 2001, Week 4
October 2001, Week 3
October 2001, Week 2
October 2001, Week 1
September 2001, Week 5
September 2001, Week 4
September 2001, Week 3
September 2001, Week 2
September 2001, Week 1
August 2001, Week 5
August 2001, Week 4
August 2001, Week 3
August 2001, Week 2
August 2001, Week 1
July 2001, Week 5
July 2001, Week 4
July 2001, Week 3
July 2001, Week 2
July 2001, Week 1
June 2001, Week 5
June 2001, Week 4
June 2001, Week 3
June 2001, Week 2
June 2001, Week 1
May 2001, Week 5
May 2001, Week 4
May 2001, Week 3
May 2001, Week 2
May 2001, Week 1
April 2001, Week 5
April 2001, Week 4
April 2001, Week 3
April 2001, Week 2
April 2001, Week 1
March 2001, Week 5
March 2001, Week 4
March 2001, Week 3
March 2001, Week 2
March 2001, Week 1
February 2001, Week 4
February 2001, Week 3
February 2001, Week 2
February 2001, Week 1
January 2001, Week 5
January 2001, Week 4
January 2001, Week 3
January 2001, Week 2
January 2001, Week 1
December 2000, Week 5
December 2000, Week 4
December 2000, Week 3
December 2000, Week 2
December 2000, Week 1
November 2000, Week 5
November 2000, Week 4
November 2000, Week 3
November 2000, Week 2
November 2000, Week 1
October 2000, Week 5
October 2000, Week 4
October 2000, Week 3
October 2000, Week 2
October 2000, Week 1
September 2000, Week 5
September 2000, Week 4
September 2000, Week 3
September 2000, Week 2
September 2000, Week 1
August 2000, Week 5
August 2000, Week 4
August 2000, Week 3
August 2000, Week 2
August 2000, Week 1
July 2000, Week 5
July 2000, Week 4
July 2000, Week 3
July 2000, Week 2
July 2000, Week 1
June 2000, Week 5
June 2000, Week 4
June 2000, Week 3
June 2000, Week 2
June 2000, Week 1
May 2000, Week 5
May 2000, Week 4
May 2000, Week 3
May 2000, Week 2
May 2000, Week 1
April 2000, Week 5
April 2000, Week 4
April 2000, Week 3
April 2000, Week 2
April 2000, Week 1
March 2000, Week 5
March 2000, Week 4
March 2000, Week 3
March 2000, Week 2
March 2000, Week 1
February 2000, Week 5
February 2000, Week 4
February 2000, Week 3
February 2000, Week 2
February 2000, Week 1
January 2000, Week 5
January 2000, Week 4
January 2000, Week 3
January 2000, Week 2
January 2000, Week 1
December 1999, Week 5
December 1999, Week 4
December 1999, Week 3
December 1999, Week 2
December 1999, Week 1
November 1999, Week 5
November 1999, Week 4
November 1999, Week 3
November 1999, Week 2
November 1999, Week 1
October 1999, Week 5
October 1999, Week 4
October 1999, Week 3
October 1999, Week 2
October 1999, Week 1
September 1999, Week 5
September 1999, Week 4
September 1999, Week 3
September 1999, Week 2
September 1999, Week 1
August 1999, Week 5
August 1999, Week 4
August 1999, Week 3
August 1999, Week 2
August 1999, Week 1
July 1999, Week 5
July 1999, Week 4
July 1999, Week 3
July 1999, Week 2
July 1999, Week 1
June 1999, Week 5
June 1999, Week 4
June 1999, Week 3
June 1999, Week 2
June 1999, Week 1
May 1999, Week 4
May 1999, Week 3
May 1999, Week 2
May 1999, Week 1
April 1999, Week 5
April 1999, Week 4
April 1999, Week 3
April 1999, Week 2
April 1999, Week 1
March 1999, Week 5
March 1999, Week 4
March 1999, Week 3
March 1999, Week 2
March 1999, Week 1
February 1999, Week 4
February 1999, Week 3
February 1999, Week 2
February 1999, Week 1
January 1999, Week 5
January 1999, Week 4
January 1999, Week 3
January 1999, Week 2
January 1999, Week 1
December 1998, Week 5
December 1998, Week 3
December 1998, Week 2
December 1998, Week 1
November 1998, Week 5
November 1998, Week 4
November 1998, Week 3
November 1998, Week 2
November 1998, Week 1
October 1998, Week 5
October 1998, Week 4
October 1998, Week 3
October 1998, Week 2
October 1998, Week 1
September 1998, Week 4
September 1998, Week 3
September 1998, Week 2
September 1998, Week 1
August 1998, Week 5
August 1998, Week 4
August 1998, Week 3
August 1998, Week 2
August 1998, Week 1
July 1998, Week 5
July 1998, Week 4
July 1998, Week 3
July 1998, Week 2
July 1998, Week 1
June 1998, Week 4
June 1998, Week 3
June 1998, Week 2
June 1998, Week 1
May 1998, Week 4
May 1998, Week 3
May 1998, Week 2
May 1998, Week 1
April 1998, Week 5
April 1998, Week 4
April 1998, Week 3
April 1998, Week 2
April 1998, Week 1
March 1998, Week 5
March 1998, Week 4
March 1998, Week 3
March 1998, Week 2
March 1998, Week 1
February 1998, Week 2
February 1998, Week 1
January 1998, Week 4
January 1998, Week 3
January 1998, Week 2
January 1998, Week 1
December 1997, Week 4
December 1997, Week 3
December 1997, Week 2
December 1997, Week 1
November 1997, Week 4
November 1997, Week 2
October 1997, Week 3
September 1997, Week 5
September 1997, Week 4
September 1997, Week 3
September 1997, Week 1
August 1997, Week 4
August 1997, Week 2
August 1997, Week 1
July 1997, Week 4
July 1997, Week 3
July 1997, Week 2
June 1997, Week 5
June 1997, Week 4
June 1997, Week 2
June 1997, Week 1
May 1997, Week 5
May 1997, Week 4
May 1997, Week 3
April 1997, Week 5
April 1997, Week 4
April 1997, Week 3
April 1997, Week 2
April 1997, Week 1
March 1997, Week 4
March 1997, Week 3
March 1997, Week 2
March 1997, Week 1
February 1997, Week 4
February 1997, Week 3
February 1997, Week 2
February 1997, Week 1
January 1997, Week 5

ATOM RSS1 RSS2



LISTSERV.ACM.ORG

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager