> What of sort of evaluation was it? Personal study of other sites? With
> colleagues? User-based feedback from surveys, focus groups, etc?
> Quantitative? Qualitative?
We've done a variety of approaches, most commonly formative evaluation/beta
testing--having people (usually kids) go through a Web activity that's 90%
complete and observing their behavior--what do they click on, what do they
skip over, what gives them pause? We seem to learn most by watching, though
we do chat with them afterwards. Usability is our top concern with this type
of testing, and it's invaluable. (Usability guru Jakob Nielsen says you only
need to test with 5 users to find 80% of usability problems:
http://www.useit.com/alertbox/20000319.html)

Along the same lines, we have a mailing list of teachers and others who have
volunteered to test new projects, and they also provide valuable feedback.

We've also done front-end focus groups, which are certainly helpful, though
you do need to be a little wary--people can spout off ideas that sound good
but which they would in fact hate if they ran into it on an actual Web site.
As Jakob Nielson says, "First Rule of Usability? Don't Listen to Users"
http://www.useit.com/alertbox/20010805.html

We recently conducted a research and evaluation project with Minda Borun and
Margaret Chambers that used online surveys and server statistics to explore
user preferences for types of online learning experiences. We collected both
quantitative and qualitative data. The research report focuses on the
quantitative, but the qualitative is equally interesting, though mainly from
an evaluation perspective. You can read the report here:
http://www.eduweb.com/likelearn_abstract.html

> Was your method satisfactory? Was it resource intensive?
Having a few kids look at beta sites is about quite easy, and extremely
helpful. Teacher-tester mailing lists are even easier and provide more
valuable feedback. A research study is obviously more resource intensive,
but we were pleasantly surprised with how easy it was to collect data--pop a
survey up for a day or two and get 50 responses, like dipping a cup in a
river. We did spend awhile on the survey design, trying to find an approach
that was minimally intrusive but still maximized the response rate. Fussing
with the survey and the scripts that control the survey window and process
the data took longer than I'd (perhaps naively) anticipated, as did prepping
the server logs for analysis (partly because we were getting logs from five
different servers, each with its own log format). However, once you have all
that set up, you can easily and quickly try out new surveys or analyze or
re-analyze server logs, so it definitely has proved worthwhile.

If you have any questions about all this, please send them on.

Cheers,
Dave Schaller

___________________________________________________________________
David T. Schaller                                  [log in to unmask]
Partner                                       http://www.eduweb.com
Educational Web Adventures                             651-641-7566

Immersive, interactive, and in-depth adventures about art, science
and history. Best of the Web: Museums & the Web 2000 and 2001

=========================================================
Important Subscriber Information:

The Museum-L FAQ file is located at http://www.finalchapter.com/museum-l-faq/ . You may obtain detailed information about the listserv commands by sending a one line e-mail message to [log in to unmask] . The body of the message should read "help" (without the quotes).

If you decide to leave Museum-L, please send a one line e-mail message to [log in to unmask] . The body of the message should read "Signoff Museum-L" (without the quotes).