Adrian,

    I worked with the education director of a small Natural history museum on developing and conducting an evaluation of their group tours. The list was very helpful when I appealed for advice. The responses are listed below. We ended up with a one page evaluation given to teachers at the start of the tour which asked questions regarding tour content (did it meet 
their curricular objectives sorts of Qs), structure (an appropriate mix of hands-on and discussion Qs) and the affective experience (do you think the tour got your students excited about science, how do you know etc.). Teachers were more than happy to fill out the form and as added incentive we added a raffle ticket that would enter their school for a chance to win a free museum tour.  The evaluation is ongoing, host docents give them to teachers when they arrive at the museum.
I hope this helps,
Rachel

Thank you everybody for your extremely helpful comments on conducting
evaluations of museum programs . Several of you responded off-list, and
several people requested the suggestions themselves, so here is the
original posting and the advice people offered (since people responded off
list, I did not include their names, for privacy, I do not mean to not give
credit where credit is due)I am pleased to say we will begin evaluating
next week!


 My apologies for cross-listings...

   I am working with the education director of a small natural history
museum to get an evaluation procedure in place of their group programs. The
programs were  developed for student groups (k-12) that come to the museum
on field trips. They are led by docents, focusing on specific topics such
as local wildlife and  prehistoric earth. I am developing a survey for the
teachers that bring their students to assess whether the museum programs
meet their objectives, how the programs could be improved and to find out
what is working already. I have gathered a fair amount of literature on
evaluations and visitor surveys, and now I am appealing to the vast
resource of the lists:
Does anyone have a similar survey/evaluation that they have had success
with and would be willing to send me? This is my first attempt at such a
project and I am curious about they breadth of material addressed in survey
questions (and what I may have missed!)
I am finding reported return rates to be quite low, teachers: is there a
threshold length after which it becomes too cumbersome to respond to such a
survey? We are considering holding a raffle for a free museum trip for a
class, entering all respondents; do such incentives usually work?
Any other words of wisdom would be greatly appreciated, you may reply on or
off list.
Thanks in advance,
Rachel Ehrenberg


If possible, definitely offer an incentive. They do work. Also, the best
return rates I have experienced are when the teacher is asked to fill out
the survey during the museum visit. In other words, don't let them take the
survey home because it will get lost among all the other stuff they have to
do. In general, with take-home surveys, I have had a 25% return rate.

The highest return rates I have experienced are when we have distributed the
surveys at the beginning of the museum program and told teachers that they
would receive a gift when they returned the survey to the museum store. The
gift can be something minimal like a poster or discount coupon for the store
or, like you suggested, a raffle ticket for a free visit.

In my experience, self-addressed and stamped envelopes do not make a
difference when trying to get teachers to return things so don't waste too
much money on stamps. Strike while the iron is hot and get the teachers to
complete the survey on site.
Good luck!

    AND:
I have years of experience generating evaluations and am still learning the
pitfalls. I have a few suggestions for you.
It is practically impossible to get evaluations returned once they have left
your museum, even if you give an sas envelope with them. The only way to
improve this kind of delivery is to put the evaluation, and the envelope if
possible, on a radically different color of paper so that it doesn't get
lost (forgotten) on a desk top.
I never found incentives, deadlines or threats to be effective in improving
response rate.

What I recommend is that you hand out and collect evaluations from the
students and the teachers before they leave your museum. The kind of
information you could get from kids is different and as important in your
research.
Make time and space for them to fill the evaluation out at the museum and
stand at the bus as they get on, take the form and shake each hand thanking
them for coming. Schedule the time, including it in the time figure you
share with them before they arrive so that the buses aren't necessarily
waiting for them. Give fifteen or twenty min. and have your docent hang by
to supervise the students so that the teachers can actually concentrate on
what they are writing.
The evaluation should have only 3 or 4 questions with lots of space
in-between the questions to invite lengthy response.
If you create student and teacher forms on the same color paper, have a
checkbox at the top to indicate it outright, for your convenience.
Do not use the score system of 1 to 5 or poor to good.
Questions that can be answered with a yes or no will be.
If you want to know something specific (about a display or speaker) you have
to ask the question specifically. We covered many areas by listing them and
giving a checkmark space, indicating if the visitor would suggest for a
friend to see or hear that feature.
For teachers ask if they have suggestions for your program to improve their
next visit.
For students ask what was the most interesting thing they learned or what
was their favorite animal.
If the students are too young to fill out a form, provide paper and crayons
to have them make a picture of their favorite thing at the museum.
Say thank you at the end of the evaluation.

I think that's about all except if you want a quotable quote for marketing
you need to phrase the question in a way that work generate such a comment.
i.e. "Write a sentence as if you were telling a friend about what you
enjoyed on your field trip."

Good luck!

AND:
I think you might get better returns and closer observations if you
asked teachers to carry around a clipboard with the quetsions when they
were in the museum. Ask them, for example, to write snippets of kids
conversations that show their interests, their questions.

AND:
 The response rate can be a problem. Ways to maximize the
response are:
    1. Give out with a postage-paid envelope so they can complete later and
return at   no cost to them.
    2. Convince them that their response matters by saying so on your
instrument  and acting on their response (they will know if things are
different next time).
    3.  Don't ask open-ended questions, precode as much as possible so it's
easy to     complete.  You can ask at the end for comments, but try to
avoid
getting them to     write lengthy thoughts.  Precoding means you know what
your after, they get    finished fast, and its easier for you to analyze
after.
    4.  Make it confidential - don't ask for their name unless it's
absolutely needed.
    5. Don't ask anything you already know - when and why they came, for
example.    You can always add indentifiers to the forms if you need to
track
replies     individually.
    6. Make it short - going over a page will reduce the response rate,
over
three   pages and you are loosing even more.
    7. I haven't found the chance to win a trip for completing a
questionnaire, much of  a draw for teachers, because they don't know their
odds of winning. I offer    individual incentives for focus group or
individual interviews (in the form of free  family admissions).
    8.  Do it another way - use volunteers to ask subsets of questions of
teachers    some time during their trip or do a telephone interview. We do
surveys, one on     one interviews, and focus groups -- all for different
reasons -- and all can be   useful.

    Hope this is of some help.

AND:
In response to your ASTC request, my organization surveys teachers (not
about
museum visits) and has found that they respond at least with a 50% return
rate if we include a $1 bill in the survey and if the survey is not all
open-ended questions.  If you can specify check-off responses at all, you
should do so.

AND:
I cannot comment on the success of survey return rates, but we get a lot of
school groups from around the nation.
Being a science centre in Australia's capital (Canberra), we rely on getting
feedback by post or phone.
We are trialling a system where teachers can post their feedback forms to us
and they go in a draw every couple of months for an item from our science
shop. I am not sure how this is going, but offering a return trip to our
science centre may not be taken up by those travelling long distances and is
less of an incentive.

AND:
As part of the 1999/2000 MSc course in Communicating
Science run at Techniquest (UK) we (the students that year)
did a small survey among teachers.

We sent all schools that were going to visit Techniquest
that week a letter in advance telling them that we were
going to interview them. We asked the schools that did not
want to participate to mention it at the desk when they
arrived.

During the free part of the visit when the children where
running around in the exhibition we asked the teachers five
short questions (sorry I don't remember them). I think that
only one school refused to participate. It is amazing how
much you can learn in only a short period of time. I
suggest that you don't miss the "anything else question".

Good Luck

AND:
You may find, as I did, that the book by Judy Diamond described below is
extremely valuable, clear, and to-the-point.  It can be obtained through
the
American Association of Museums Bookstore
http://www.aam-us.org/text/bookstore.htm

Here is a copy of their description of it.
PRACTICAL EVALUATION GUIDE: TOOLS FOR MUSEUMS AND OTHER
        INFORMAL EDUCATIONAL SETTINGS
        Judy Diamond
        AltaMira Press/American Association for State and Local History

        Visitor evaluations provide clues to the effectiveness of exhibits
and
programs and provide
        insights into how people learn in informal educational settings.
This
title presents the basic
        principles and techniques needed to design, implement, and present
an
evaluation project. It is
        a clearly and simply written guide with sample questionnaires that
should prove an invaluable
        resource for competent, reliable evaluations in informal
educational
settings of any kind.

        PAPER 192PP. 1999 · ISBN 0-7619-8940-4 · $ 22.00

Best wishes for your success in this project.

***********************************************************************
More information about the Informal Science Education Network and the
Association of Science-Technology Centers may be found at http://www.astc.org.
To remove your e-mail address from the ISEN-ASTC-L listserv, send the
text SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]


--On Tuesday, January 9, 2001 2:36 PM -0600 "Perez, Adrian" <[log in to unmask]> wrote:

> Good People:
>
> My department, which conducts classes (grades K-12) both on and off site,
> is beginning the process of self-evaluation to determine the
> effectiveness of our programming.  Due to budgetary constraints we cannot
> currently afford to hire an outside consultant and so are developing this
> self-evaluation in-house.  Eventually we would like to hire an outside
> firm to participate but for now we are fumbling ahead using our instincts
> and crossed fingers as guides.  We have an excellent person within our
> organization who is guiding us, but I also wanted to send out an
> information request to fellow listers.
>
> For those of you who have educational programs (heck, ANY kind of program)
> and have evaluated/are evaluating/will evaluate them, i humbly pose these
> questions:
>
> * What kind of programming do you do?
>
> * Do you perform any evaluations of said programming?
>
> *Was your evaluation developed from an outside contractor or internally?
>
> *How long has the evaluation been in progress (or will it be in progress)?
>
> *What types of methods are being utilized to perform the evaluation?
>
> As I would like to develop a dialogue with respondents, please reply to me
> directly.  Thanks in advance to everyone.  Believe me, any information no
> matter how orbital/tangential would be greatly appreciated.
>
> Yours,
> Adrian
>
> _____________________________
>
> Adrian Perez
> Manager of School and Outreach Programs
> John G. Shedd Aquarium
> e: [log in to unmask]
> p: 312.692.3166
> f: 312.663.1472
>
> =========================================================
> Important Subscriber Information:
>
> The Museum-L FAQ file is located at
> http://www.finalchapter.com/museum-l-faq/ . You may obtain detailed
> information about the listserv commands by sending a one line e-mail
> message to [log in to unmask] . The body of the message should
> read "help" (without the quotes).
>
> If you decide to leave Museum-L, please send a one line e-mail message to
> [log in to unmask] . The body of the message should read
> "Signoff Museum-L" (without the quotes).

========================================================Important Subscriber Information:

The Museum-L FAQ file is located at http://www.finalchapter.com/museum-l-faq/ . You may obtain detailed information about the listserv commands by sending a one line e-mail message to [log in to unmask] . The body of the message should read "help" (without the quotes).

If you decide to leave Museum-L, please send a one line e-mail message to [log in to unmask] . The body of the message should read "Signoff Museum-L" (without the quotes).