CHI-WEB Archives

ACM SIGCHI WWW Human Factors (Open Discussion)


Options: Use Forum View

Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Toby Biddle <[log in to unmask]>
Reply To:
Toby Biddle <[log in to unmask]>
Thu, 30 Sep 2010 09:33:47 +1000
text/plain (90 lines)
Hi Kay,

Our tool, Loop11 ( can and has been used for A/B testing by many of our customers before. We've also written a case study ( that outlines how a project was set up by one of our customers.  I also agree with Susan's response in which she suggests that inviting careful perusal of the differences doesn't encourage natural behaviour. 

Creating task-based scenarios to see which design performs the best is the better way to run A/B testing. If you haven't already signed up to Loop11 feel free to do so. Your first project is free, so you can use it to see whether A/B testing will work for you.

I hope this helps.

Toby Biddle
t: (03) 9684 3470
m: 0402 113 104
f: (03) 9684 3434
e: [log in to unmask]
skype: toby.biddle

119 Ferrars Street, South Melbourne, Victoria, 3205 

The information contained in this e-mail and any files transmitted with it is private and confidential. It is intended for the named addressee only. If you are not the intended addressee you are prohibited from storing, copying or using the information in any way. If you received this e-mail due to a transmission error please notify the sender immediately. No liability is accepted by UsabilityOne for any losses caused by viruses contracted during transit over the Internet or present in any receiving system. This e-mail is not intended to create legally binding commitments on behalf of UsabilityOne, nor do its contents reflect the corporate views or policies of UsabilityOne. 

-----Original Message-----
From: ACM SIGCHI WWW Human Factors (Open Discussion) [mailto:[log in to unmask]] On Behalf Of Susan Price
Sent: Thursday, 30 September 2010 12:18 AM
To: [log in to unmask]
Subject: Re: Usability test design approaches for A/B testing


Your proposed test is not what I think of as A/B testing. Our A/B testing is showing one or the other variant to the same user segment (usually in the live production environment) and watching which performs better.

Presenting on the same screen side by side is an attempt to not lead the participants? This seems to invite careful perusal of the differences, and that's usually not what we're going for. What we seek in the usability lab are the untutored, knee-jerk reactions that we can't anticipate. Inviting careful perusal invites participants to play "designer." Our team doesn't suffer from a lack of such opinions - what we seek are the revelations that we weren't able to anticipate.

There will be unavoidable left-right bias I believe, and the test experience doesn't compare to the "one solution per screen" actual experience. 

Seems like it might be better to show each variant (and perhaps some dummies) several times and have the participant rank each with a numerical scale, with duplications of each to weed out the biases.


On Sep 29, 2010, at 7:54 AM, Kay Corry Aubrey wrote:

> Hi - can anyone on the  Chi-web list suggest innovative usability test
> design approaches for A/B testing? Both A and B will be presented on the
> same screen and the user needs to peruse each and choose the one they
> prefer. I am looking for ideas on the best UIs to present text and pictures
> most effectively for this type of purpose and was wondering if folks on the
> list have suggestions. 
> Please send your ideas to me and I will collect, summarize, and send out to
> the list. 
> Thanks!
> Kay 
> Kay Corry Aubrey, user-centered research and design 
> Usability Resources Inc | |
> [log in to unmask]
> Phone: 781-275-3020 | Fax: 1-781-998-0325
>    --------------------------------------------------------------
>    Tip of the Day: Suspend your subscription if using auto replies
>            CHI-WEB:
>              MODERATOR: mailto:[log in to unmask]
>    --------------------------------------------------------------

    Tip of the Day: Suspend your subscription if using auto replies
              MODERATOR: mailto:[log in to unmask]

    Tip of the Day: Suspend your subscription if using auto replies
              MODERATOR: mailto:[log in to unmask]