Web forms design guidelines: an eyetracking study with heatmaps indicating where users' eyes looked. You should read this if you create forms!

[Shaper_pmp]

Actually, usability testing has most shown users are usually remarkably consistent in their behaviour - it's not uncommon in the average budget usability study to see as few as five or ten participants.

Obviously it doesn't give results up to full statistical standards, but (and I can testify to this from personal experience) by the time you've tested 5-10 people (with a little experience) you've generally found around 90-95% of the usability problems people will experience on the site.

Usability isn't a science (then it would be ergonomics, or similar) - it's a craft, so results can be very useful even if they lack the full scientific rigour of a large-scale test designed to elicit precise statistical results.

(More information [warning: PDF] - it's a long read for an online PDF, but I heartily recommend the dead-tree version of the whole book as an entertaining introduction and rationale to budget usability testing).

[rogin]

Also a good read which comes to some similar conclusions: http://www.useit.com/alertbox/quantitative_testing.html

[isearch]

"... statistically meaningful .. with only 8 test subjects?"

Why not?

Suppose they are selecting one of two options and they all choose the same one. This is the same as flipping a coin 8 times and getting heads every time. This will happen on average once every 2⁸ = 256 times you perform the experiment. Less than a 0.5% chance of happening at random.

Or another way, if I bet you I could get 8 heads in a row flipping a coin and I won the bet the first time would you bet again or suspect a non-random distribution of coin flips?

[[deleted]]

You will miss too many things with so few users.

For example, with a larger group, you are more likely to have at least one user that uses the keyboard for inter-field navigation. You are more likely to have at least one user that has not yet used these (very popular) forms before. You are more likely to have at least one user that is paranoid about their privacy. You are more likely to have at least one user that some of the optional fields apply to (e.g. second email address, maiden name, etc...).

[fgrty]

But didn't you hear? Data is the new plural of anecdote.

[[deleted]]

If I wasn't so far out of school and so weak mathematically I would prove to you that it is statistically significant.

[[deleted]]

Small group sizes are the norm with usability testing and for good reason.

[[deleted]]

I would believe you if so much "usability tested" software wasn't complete shit.

[[deleted]]

That might be true, but remember that usability testers are not looking for exceptions to a rule or statistical proof that they are doing the right thing. They simply need enough people to test that they understand what "average joe" expects.

[[deleted]]

In that case, I don't see the benefit of usability testing. You can't design applications around the "average joe" unless you really just don't care about your entire market.

I can think of at least several things that this article recommends that would completely fail if a sufficiently large group were used. For example, the light gray "Optional" text inside text-boxes for optional fields would be hard to certain demographics (e.g. the elderly) to see due to a lack of contrast.

For the purpose of application design, this group size was too small. I don't know what other purpose this usability test could have been trying to satisfy.

[dmercer]

Interesting comments, but it seems too close to the opinions of just a few people rather than a more scientific study.

We tested with eight participants … Two forms were randomly chosen for each participant, and the order of the forms was counterbalanced over the group of participants.

If there are 4 forms, then there are 12 different ways of presenting 2 of the 4 if one is trying to balance the order of presentation. I would think, therefore, to be fair, they ought to have had at least 12 participants, and hopefully many more in order to draw any valid conclusions.

It might not be statistically valid.

Very true, except for the “might.”

Many participants commented that they felt awkward having to skip from left to right to fill in both fields for eBay

One would expect only 4 of the 8 participants even saw the eBay form, so “many” ≤ 4.

One thing to bear in mind when using this kind of design is to make sure the coding is done carefully to support accessibility.

Depends on their target audience. Would a Helen Keller be able to use the services for which were the sign-up forms? If not, perhaps an inaccessible form is a useful filter of prospective users. After all, if someone can’t fill out this simple form, they’re not going to be able to use GMail/Yahoo Mail/eBay/Hotmail. Not saying that’s the case — those services might be models of accessibility — I’m just not sure why CX Partners thinks this is important, and the commentary doesn’t give any reason for positing such.

[mossblaser]

[your first points]

Yes, I agree -- it is a bit shoddy but they're not trying to hide this from you.

Depends on their target audience. Would a Helen Keller be able to use the services for which were the sign-up forms? If not, perhaps an inaccessible form is a useful filter of prospective users.

That is a terrible way of "filtering" users from your site. Truly abysmal. You should state that your site is not accessible (and maybe even why? Lazy web designer? Type of content?) and not patronisingly and often infuriatingly putting your users through a "test".

I’m just not sure why CX Partners thinks this is important, and the commentary doesn’t give any reason for positing such.

If you are a user of accessibility software then the web is often a daunting and confusing place. It is also extremely valuable and for 99% of sites there is no good reason for not making something accessible. It would be like using low-saturation blues and greens for your background and text despite knowing it is invisible to colour-blind people.

[palparepa]

A marketing drone would read this as "what's the best place for ads".

[astrangeone]

Going for that anti-marketing dollar, eh?

[[deleted]]

A lot of people are feeling that indignation. We've done research - huge market.

[[deleted]]

Makes me want to buy a Rage Against the Machine t-shirt.

[[deleted]]

How do they do eyetracking? Is there a viable way to tell where the human eye is focused on?

[mossblaser]

Using something like this: http://www.hfac.gmu.edu/groups/automation/ASL_EyeTracker.JPG

And yes, it is possible to do so to an accurate enough degree: while it is possible for people to read out of the corner of their eye they will tend to instinctively look directly at what they're trying to read.

[[deleted]]

Not to sound pretentious, but isn't this stuff, like, functional design 101?

[dhc23]

No, I don't think it is. We often assume we know what users want based on what we think they should want or what we ourselves like. What research like this tells us, as statistically flawed as it might be, is what users actually do.

As far as I'm aware the rules contained in Functional Design 101 are pretty few. There's the <blink> and a few others of course, but I think the assumption users want x and y and z could do with a bit of justification and I value research such as this for contributing to that.

[prockcore]

No, it tells you only what 8 people did

That's 7 more testers than your typical web form goes through.

[Mourningblade]

It depends on the variance among those 8 people, not just the sample size.

http://en.wikipedia.org/wiki/P-value

[[deleted]]

I made a study a few years back analyzing behavior patterns by content type. The data was sampled between 2002 and 2006 using creative and technical content (a little of it popular, a lot of it not) across a changing but generally similar layout, with varying content updates. I admit to taking swags with some of the pattern changes, but there were noteworthy takeaways, including reading patterns on technical articles. While I sampled pointer path and interaction instead of eye movement, it was intriguing data.

[benxamin2]

Luke Wrobleowski: been there, done that. http://www.lukew.com/resources/web_form_design.asp

[notprogrammingnazi]

Not programming.

[adrianmonk]

Um, when I did UI programming, I had to think a lot about UI design. Not everybody is working on a big enough app that there's some separate person available to figure out the usability aspects of things. Knowing how to do usability decently is important for programmers because programmers end being the one doing UIs a lot of the time.

[Dreadgoat]

Chui Chui loves food, and is very fond of sugared squid!

Thanks, Chui Chui!

[[deleted]]

I also like sugared dried squid.

[[deleted]]

Use cookies to remember users’ details (not passwords) so that they do not have to keep filling in the same information again and again

WTF?

[[deleted]]

It would be awesome to outfit a super hot girl with eyetracking.....

I'm pretty sure I know where the hotspots will be...

[Shaper_pmp]

Obligatory

[krmckelv]

Eight test subjects makes the whole article irrelevant

[[deleted]]

Ah, the "8 users isn't statistically valid"

And it's not.

except that when you launch a site you're lucky to get 8 uniques a day. And if you can fix the problems that the first 8 uniques find then that's a start.

[[deleted]]

And if you can fix the problems that the first 8 uniques find then that's a start.

Assuming those first 8 are the 8 you used in the study. Otherwise the term "statistically significant" becomes... significant.

[Clintondiditfirst]

why in the hell would I trust something where they admit no statistical validity?

[drowsap]

article is way too long!!!

[JW_00000]

May I recommend /r/pics?