Adrian, I worked with the education director of a small Natural history museum on developing and conducting an evaluation of their group tours. The list was very helpful when I appealed for advice. The responses are listed below. We ended up with a one page evaluation given to teachers at the start of the tour which asked questions regarding tour content (did it meet their curricular objectives sorts of Qs), structure (an appropriate mix of hands-on and discussion Qs) and the affective experience (do you think the tour got your students excited about science, how do you know etc.). Teachers were more than happy to fill out the form and as added incentive we added a raffle ticket that would enter their school for a chance to win a free museum tour. The evaluation is ongoing, host docents give them to teachers when they arrive at the museum. I hope this helps, Rachel Thank you everybody for your extremely helpful comments on conducting evaluations of museum programs . Several of you responded off-list, and several people requested the suggestions themselves, so here is the original posting and the advice people offered (since people responded off list, I did not include their names, for privacy, I do not mean to not give credit where credit is due)I am pleased to say we will begin evaluating next week! My apologies for cross-listings... I am working with the education director of a small natural history museum to get an evaluation procedure in place of their group programs. The programs were developed for student groups (k-12) that come to the museum on field trips. They are led by docents, focusing on specific topics such as local wildlife and prehistoric earth. I am developing a survey for the teachers that bring their students to assess whether the museum programs meet their objectives, how the programs could be improved and to find out what is working already. I have gathered a fair amount of literature on evaluations and visitor surveys, and now I am appealing to the vast resource of the lists: Does anyone have a similar survey/evaluation that they have had success with and would be willing to send me? This is my first attempt at such a project and I am curious about they breadth of material addressed in survey questions (and what I may have missed!) I am finding reported return rates to be quite low, teachers: is there a threshold length after which it becomes too cumbersome to respond to such a survey? We are considering holding a raffle for a free museum trip for a class, entering all respondents; do such incentives usually work? Any other words of wisdom would be greatly appreciated, you may reply on or off list. Thanks in advance, Rachel Ehrenberg If possible, definitely offer an incentive. They do work. Also, the best return rates I have experienced are when the teacher is asked to fill out the survey during the museum visit. In other words, don't let them take the survey home because it will get lost among all the other stuff they have to do. In general, with take-home surveys, I have had a 25% return rate. The highest return rates I have experienced are when we have distributed the surveys at the beginning of the museum program and told teachers that they would receive a gift when they returned the survey to the museum store. The gift can be something minimal like a poster or discount coupon for the store or, like you suggested, a raffle ticket for a free visit. In my experience, self-addressed and stamped envelopes do not make a difference when trying to get teachers to return things so don't waste too much money on stamps. Strike while the iron is hot and get the teachers to complete the survey on site. Good luck! AND: I have years of experience generating evaluations and am still learning the pitfalls. I have a few suggestions for you. It is practically impossible to get evaluations returned once they have left your museum, even if you give an sas envelope with them. The only way to improve this kind of delivery is to put the evaluation, and the envelope if possible, on a radically different color of paper so that it doesn't get lost (forgotten) on a desk top. I never found incentives, deadlines or threats to be effective in improving response rate. What I recommend is that you hand out and collect evaluations from the students and the teachers before they leave your museum. The kind of information you could get from kids is different and as important in your research. Make time and space for them to fill the evaluation out at the museum and stand at the bus as they get on, take the form and shake each hand thanking them for coming. Schedule the time, including it in the time figure you share with them before they arrive so that the buses aren't necessarily waiting for them. Give fifteen or twenty min. and have your docent hang by to supervise the students so that the teachers can actually concentrate on what they are writing. The evaluation should have only 3 or 4 questions with lots of space in-between the questions to invite lengthy response. If you create student and teacher forms on the same color paper, have a checkbox at the top to indicate it outright, for your convenience. Do not use the score system of 1 to 5 or poor to good. Questions that can be answered with a yes or no will be. If you want to know something specific (about a display or speaker) you have to ask the question specifically. We covered many areas by listing them and giving a checkmark space, indicating if the visitor would suggest for a friend to see or hear that feature. For teachers ask if they have suggestions for your program to improve their next visit. For students ask what was the most interesting thing they learned or what was their favorite animal. If the students are too young to fill out a form, provide paper and crayons to have them make a picture of their favorite thing at the museum. Say thank you at the end of the evaluation. I think that's about all except if you want a quotable quote for marketing you need to phrase the question in a way that work generate such a comment. i.e. "Write a sentence as if you were telling a friend about what you enjoyed on your field trip." Good luck! AND: I think you might get better returns and closer observations if you asked teachers to carry around a clipboard with the quetsions when they were in the museum. Ask them, for example, to write snippets of kids conversations that show their interests, their questions. AND: The response rate can be a problem. Ways to maximize the response are: 1. Give out with a postage-paid envelope so they can complete later and return at no cost to them. 2. Convince them that their response matters by saying so on your instrument and acting on their response (they will know if things are different next time). 3. Don't ask open-ended questions, precode as much as possible so it's easy to complete. You can ask at the end for comments, but try to avoid getting them to write lengthy thoughts. Precoding means you know what your after, they get finished fast, and its easier for you to analyze after. 4. Make it confidential - don't ask for their name unless it's absolutely needed. 5. Don't ask anything you already know - when and why they came, for example. You can always add indentifiers to the forms if you need to track replies individually. 6. Make it short - going over a page will reduce the response rate, over three pages and you are loosing even more. 7. I haven't found the chance to win a trip for completing a questionnaire, much of a draw for teachers, because they don't know their odds of winning. I offer individual incentives for focus group or individual interviews (in the form of free family admissions). 8. Do it another way - use volunteers to ask subsets of questions of teachers some time during their trip or do a telephone interview. We do surveys, one on one interviews, and focus groups -- all for different reasons -- and all can be useful. Hope this is of some help. AND: In response to your ASTC request, my organization surveys teachers (not about museum visits) and has found that they respond at least with a 50% return rate if we include a $1 bill in the survey and if the survey is not all open-ended questions. If you can specify check-off responses at all, you should do so. AND: I cannot comment on the success of survey return rates, but we get a lot of school groups from around the nation. Being a science centre in Australia's capital (Canberra), we rely on getting feedback by post or phone. We are trialling a system where teachers can post their feedback forms to us and they go in a draw every couple of months for an item from our science shop. I am not sure how this is going, but offering a return trip to our science centre may not be taken up by those travelling long distances and is less of an incentive. AND: As part of the 1999/2000 MSc course in Communicating Science run at Techniquest (UK) we (the students that year) did a small survey among teachers. We sent all schools that were going to visit Techniquest that week a letter in advance telling them that we were going to interview them. We asked the schools that did not want to participate to mention it at the desk when they arrived. During the free part of the visit when the children where running around in the exhibition we asked the teachers five short questions (sorry I don't remember them). I think that only one school refused to participate. It is amazing how much you can learn in only a short period of time. I suggest that you don't miss the "anything else question". Good Luck AND: You may find, as I did, that the book by Judy Diamond described below is extremely valuable, clear, and to-the-point. It can be obtained through the American Association of Museums Bookstore http://www.aam-us.org/text/bookstore.htm Here is a copy of their description of it. PRACTICAL EVALUATION GUIDE: TOOLS FOR MUSEUMS AND OTHER INFORMAL EDUCATIONAL SETTINGS Judy Diamond AltaMira Press/American Association for State and Local History Visitor evaluations provide clues to the effectiveness of exhibits and programs and provide insights into how people learn in informal educational settings. This title presents the basic principles and techniques needed to design, implement, and present an evaluation project. It is a clearly and simply written guide with sample questionnaires that should prove an invaluable resource for competent, reliable evaluations in informal educational settings of any kind. PAPER 192PP. 1999 · ISBN 0-7619-8940-4 · $ 22.00 Best wishes for your success in this project. *********************************************************************** More information about the Informal Science Education Network and the Association of Science-Technology Centers may be found at http://www.astc.org. To remove your e-mail address from the ISEN-ASTC-L listserv, send the text SIGNOFF ISEN-ASTC-L in the BODY of a message to [log in to unmask] --On Tuesday, January 9, 2001 2:36 PM -0600 "Perez, Adrian" <[log in to unmask]> wrote: > Good People: > > My department, which conducts classes (grades K-12) both on and off site, > is beginning the process of self-evaluation to determine the > effectiveness of our programming. Due to budgetary constraints we cannot > currently afford to hire an outside consultant and so are developing this > self-evaluation in-house. Eventually we would like to hire an outside > firm to participate but for now we are fumbling ahead using our instincts > and crossed fingers as guides. We have an excellent person within our > organization who is guiding us, but I also wanted to send out an > information request to fellow listers. > > For those of you who have educational programs (heck, ANY kind of program) > and have evaluated/are evaluating/will evaluate them, i humbly pose these > questions: > > * What kind of programming do you do? > > * Do you perform any evaluations of said programming? > > *Was your evaluation developed from an outside contractor or internally? > > *How long has the evaluation been in progress (or will it be in progress)? > > *What types of methods are being utilized to perform the evaluation? > > As I would like to develop a dialogue with respondents, please reply to me > directly. Thanks in advance to everyone. Believe me, any information no > matter how orbital/tangential would be greatly appreciated. > > Yours, > Adrian > > _____________________________ > > Adrian Perez > Manager of School and Outreach Programs > John G. Shedd Aquarium > e: [log in to unmask] > p: 312.692.3166 > f: 312.663.1472 > > ========================================================= > Important Subscriber Information: > > The Museum-L FAQ file is located at > http://www.finalchapter.com/museum-l-faq/ . You may obtain detailed > information about the listserv commands by sending a one line e-mail > message to [log in to unmask] . The body of the message should > read "help" (without the quotes). > > If you decide to leave Museum-L, please send a one line e-mail message to > [log in to unmask] . The body of the message should read > "Signoff Museum-L" (without the quotes). ========================================================Important Subscriber Information: The Museum-L FAQ file is located at http://www.finalchapter.com/museum-l-faq/ . You may obtain detailed information about the listserv commands by sending a one line e-mail message to [log in to unmask] . The body of the message should read "help" (without the quotes). If you decide to leave Museum-L, please send a one line e-mail message to [log in to unmask] . The body of the message should read "Signoff Museum-L" (without the quotes).