Module 3: Building an Evaluation Plan

Crafting Evaluation Questions

Evaluation questions guide the evaluation. It is often unrealistic to evaluate everything you want to explore about the program. Prioritizing evaluation questions will help focus your time and resources.  Note:  Evaluation questions refer to big, broad questions that define what you want to learn from your evaluation.  These are not necessarily the questions that you will include in your survey or evaluation tool.

TASK 1: Review page 3 of Module 2 in the Collective Outcomes Toolkit.  Refer to the list of program stakeholders you identified in Module 2: Task 4. Select 5 of the stakeholders and identify at least one question the stakeholder might have for the evaluation in your course journal (Include program participants as at least one of your stakeholders.)  

TASK 2: Select at least 1 of the stakeholders you listed above and interview them to learn more about what they would be interested in learning from the program evaluation.  Identify ways you might be able to engage other stakeholders listed above in identifying evaluation questions.  Record your reflections (~1 paragraph) in your course journal.

TASK 3:  Based on your conversation and the information that will support you in improving your program, list 3-5 evaluation questions you will be focusing on in your course journal.

Evaluation Plan

The final step in developing your evaluation is thinking through each element of the evaluation plan. Like the logic model, an evaluation plan helps provide a concise picture of your evaluation. 

TASK 4: Start working on your evaluation plan.  Read pages 4-7 of the Collective Outcomes Module 2 Toolkit.  Using your evaluation questions, fill in the outcomes (select outcomes from your logic model and the Colorado Collective Outcomes), indicators, and objectives you would like to measure in your evaluation in the table in your course journal.

TASK 5:  Do you have any data that can be re-purposed to answer your evaluation questions? (e.g., registrations, applications, attendance records, participant projects and using embedded assessments or assessments that are directly built into your programming). Don’t forget the easy data! As a part of your evaluation work plan, remember some of the "simple" indicators such as counting the participants at events or the number of bags of litter collected on river clean-up day. It can be easy to forget to collect this information when you focus on the more intensive tasks of developing and administering evaluation instruments. What existing data might you be able to use? Record possible existing data sources in your course journal.

TASK 6:  What additional data will you need? Read pages 8-12 in the Collective Outcomes Module 2 Toolkit.  What kind of data (qualitative or quantitative) will best answer your evaluation questions?  Which type of tools (surveys, interviews, etc.) might be the most useful for your program?  Record your reflections in your course journal.

TASK 7:  Time is one of the biggest barriers in collecting data.  Evaluations tend to be most successful when they are built into the program structure.  Review the Program Learning Cycle Developed by the BEETLES Project.  Are there opportunities to build evaluation into the program or learning experience reflection?  How could the evaluation process support participant learning AND provide meaningful data?  Record your thoughts and ideas in your course journal.

TASK 8:  Review the variety of sample evaluation tools available on pages 14-18 of the Collective Outcomes Module 2 Toolkit.  You can access the full tools using EITHER the database or folders included below (both contain the same information).  Select at least one of the tools and modify it for your program.  You may want to combine questions from several different tools or modify an existing tool that your organization uses to better reflect the Colorado Collective Outcomes.  Copy the tool and your modifications into your course journal.

TASK 9:  Look back to the outcomes, indicators, and objectives you drafted in Task 4.  Make any changes based on your reflections and complete the Data Collection Tools and Methods Column.  

TASK 10:  Who and when will you administer your evaluation?  Be thoughtful about who collects the data. For example, in a youth program, does it make sense for youth to help collect the data? If you’re conducting a focus group, does it make sense for a BIPOC person to lead that focus group? Does it make sense to have an environmental educator other than the one who taught the session collect feedback on the educational experience? It matters who does the asking. How often will you collect data?  Be sure to design your evaluation so that it is feasible to implement.  Complete the final column, “Staff and Timing” in your evaluation plan, in your course journal.

TASK 11:  Identify your next steps.  Do you need to modify any additional tools?  What actions do you need to take to put your evaluation plan into action?  Record your reflections in your course journal.