Program Evaluation

PADP 8640, Dr. Tyler Scott

Tuesday 6:30 to 9:15pm, Baldwin 307

Office Hours: Monday 9am-12pm or by appointment

Course list-serv: [email protected]

Overview

Broadly, policy analysis can be grouped into two categories: ex ante analysis, used to analyze policy problems generally and provide recommendations prior to implementation, and ex post analysis, used to evaluate the effectiveness of policies after implementation. This course focuses on ex post analysis, in the form of both quantitative and qualitative program evaluation, and thus complements topics covered in other courses on ex ante analysis (e.g., Policy Analysis I and II). By the end of the course you should understand the fundamentals of program evaluation from a variety of methodological perspectives, including strengths and weaknesses of quantitative and qualitative techniques, observations, case studies, focus groups, and panel studies.

The course will explore the territory of policy and program evaluation design through an experiential project, at the same time that the course addresses, in succession, the central design challenges that confront policy evaluators and ways these may be addressed. Course participants will be divided into design teams (as much as possible mixing research expertise and working experience with policy and evaluation), which will develop an evaluation design in response to an actual recent RFP. The teams will periodically issue interim design memos for peer review (by the instructor and 3-4 classmates), and a final design proposal at the end of the course. Alongside this team-based design work, individuals will carry out individual critique assignments, designed to help them recognize and analyze evaluation design issues as these arise in practical cases.

The design of this course rests on several premises about policy and program evaluation: First, it is applied, sponsored research, aimed at answering policy-related questions for some sponsoring group or client, as well as broader policy audiences. Second, policy and program evaluation is inherently a mixed methods research design problem; few, if any responsible evaluations of complex policies or programs can be done without attending to both qualitative and quantitative aspects of the evaluation problem. Third, it is an inherently political process, in which the evaluation design must navigate the often competing interests of various parties, including but not limited to those of the client. Fourth, policy evaluation is action oriented, and as such, part of the design problem involves creating viable ways of informing or guiding action (e.g., often decisions about the policy in question, but also other areas of decision-making or policy action), though not determining action. Fifth, policy evaluation is highly context-dependent, in terms of both the contexts in which the policy or program currently operates or might in the future, and those surrounding and shaping the actions that consumers of the evaluation results might take. Finally, at its root, evaluation is an educative process, and its ultimate success depends on what and how much relevant audiences and users learn from it.

Objectives

By the end of this course, students will be able to:

• Identify key issues and dimensions of a policy or program evaluation problem, and note where and how an evaluator or evaluation team can approach these issues.
• Create a viable evaluation design, in response to a stated evaluation problem as rendered by a sponsoring or client group (e.g., as in an RFP calling for an evaluation of a given policy).
• Visualize how to bring different kinds of methods (quantitative and qualitative) together in ways that can render rigorous and potentially useful answers to evaluation questions.
• Critique published examples of evaluation research AND already developed proposals for undertaking evaluation work, in such a way that the strengths and weaknesses of these designs are brought to light.
• Offer constructive feedback on colleagues' work and incorporate feedback into one's own work.
• Participate productively in team-based evaluation design, in situations where no single team member has expertise in all aspects of the design.

Prerequisites

There are no formal prerequisites for this course. That said, this course is designed with an assumption that you have a basic background in descriptive and inferential statistics and research methods more generally. I will also assume that you have the capacity to read and interpret basic descriptive and inferential statistics (including reading/interpreting professional and academic studies on public policy).

Office Hours

While I am happy to hold weekly office hours, I fully acknowledge that other classes, work schedules, childcare constraints, etc., can make it difficult to get to Baldwin Hall within a set 90 minute window. For this reason I am happy to talk on the phone with you if that is more convenient. Or, if you need to coordinate a conversation with me and your group members, I am happy to meet with you virtually using Google Meetup, Skype, or similar technology. If you have a topic you would like to bounce around or you seek a detailed amount of feedback about something related to class, my only request is that you consider a medium other than email since writing out long emails (and replies) is often less efficient than a conversation.

Readings

Readings will be made available through the course page in UGA e-learning commons. There is no required textbook for this course, but students intending to specialize in program evaluation may want to purchase a classic textbook such as Rossi, Lipsey, and Freeman. Evaluation: A Systematic Approach. 7th edition. Thousand Oaks, CA: Sage Publications, 2004 and read relevant chapters that parallel the syllabus.

Attendance

Class preparation and participation are very important for success in this course ' in fact, participation constitutes 25 percent of your grade. Please arrive on time to class and attend each class. An absence is excused if you email me in advance of the class meeting and only in the case of illness, documented emergencies, and unavoidable conflict due to official university obligations. If you anticipate missing more than two classes I encourage you to drop this class and find another elective that is more conducive to your schedule. Students who miss more than 2 classes without excuse will not receive any participation points. Job interviews and job-related conflicts are not considered excused absences. Absences reported after missing class are considered unexcused unless valid documentation is provided. I expect students who are unable to attend class to obtain class materials and notes from classmates. Missing class is not an excuse for turning in late assignments.

Collaborative Work Time

It will take some time together and good communication, as well as judicious division of labor, for your team to produce a good design, under the tight time constraints that operate in this course (and in many evaluation design situations). To facilitate that work, you can count on an hour in class devoted to the team’s planning needs, as shown in the course outline. Other than that, you may want to get together outside of the class to hash out matters that are not as easily handled in class or over email; that is up to you, and of course is dependent on the flexibility of your schedules.

Participation

As mentioned above, class preparation and participation are very important for success in this class. I ask that you attend class, arrive on time, complete assigned readings, and to contribute to class activities through active participation and involvement. Everyone benefits tremendously when there is active participation in class. Class discussions are not an empty exercise to gain points but an effort to teach each other how to engage in respectful and high-level discussions. Come to class with enthusiasm and ready to engage me, your classmates, the material, and your abilities! As you will see, I've designed participation credit for this course in a way that rewards multiple types of participation, not just speaking up in class.

To facilitate discussion in class and in small groups, I ask that you pose on at least five occasions during the semester two or three discussion questions to the class about the week's required reading material. To receive participation credit for your question(s), you will need to post the question(s) pertaining to that week's material by noon (12:00 PM) on the day of class to the course's online discussion board. The question(s) you pose about the week's required readings need not be overly complicated. At the same time, please go beyond a question that could be answered with a 'yes/no' (e.g. 'Do you agree with Merten's point about...?' does not provide sufficient material to stimulate discussion). The point of generating questions like this is to stimulate your own critical thinking about the material and provide your classmates with insight into your takeaways related to the week's required readings. When thinking about your question, you are welcome to connect your question about the week's materials to material covered in previous weeks. The class discussion board also allows you to post a reply to someone's questions. That is, if you have a question that builds on or extends another classmate's question, you are welcome to offer that in place of your own stand-alone question.

Academic Integrity

As a University of Georgia student, you have agreed to abide by the University's academic honesty policy, 'A Culture of Honesty,' and the Student Honor Code. All academic work must meet the standards described in 'A Culture of Honesty' found at: https://ovpi.uga.edu/academic-honesty/academic-honesty-policy. Lack of knowledge of the academic honesty policy is not a reasonable explanation for a violation. Questions related to course assignments and the academic honesty policy should be directed to the instructor.

Special Accommodations

If you have a learning disability, sensory or physical disability or if English is not your first language and you need special assistance in lecture, reading assignments, or written assignments, please contact the instructor at the beginning of the semester. Students with chronic conditions (illness, disability, extenuating personal or family circumstances) that may require special accommodations must notify me in writing by February 2nd. In the case of chronic illness, you must provide a doctor's note written on letterhead with the doctor's name, signature and telephone number. Excuses for chronic conditions will not be granted if documentation is not provided before the February 2nd. The physician must be located in the United States, preferably locally.

Students needing accommodations because of disability will need to register with UGA's Disability Resource Center (DRC) and complete the appropriate forms issued by DRC before accommodations can be provided. The DRC office at the University of Georgia is located at 114 Clark Howell Hall, or the DRC can be reached by phone at (706) 542-8719. For more information, please see: https://drc.uga.edu/

Working in Teams

By this point in your graduate school experience you are likely well acquainted with group work and team projects. For this course, note that 55% of your grade is tied to team deliverables.

I invite you to divide up work however you see fit over the course of the semester. For consistency's sake, however, please delegate one team member as the lead who will be responsible for submitting all your team assignments.

You are welcome to approach me at any point in the semester if you're struggling with a teammate, but I also encourage you to remedy the situation on your own to the best of your ability. Note that at the end of the semester, I will ask you to assess your contribution to the team project and the contribution of your teammates. A blaring lack of contribution may affect a team member's final project grade negatively.

Email

You are welcome to email me whenever you have a question or need clarification about something related to class. Please begin the subject line with "PADP 8640" so that I can clearly see the email among our other correspondence. Please also allow sufficient time (12-24 hours) for a response. Shane and I will do our best to reply as soon as possible but oftentimes we may be traveling, in day-long meetings, or engaged in other activities that take us away from email. If you are emailing about a group project, please follow the general convention of also copying in your teammates so as to avoid redundant correspondence. In certain cases, you may pose a good question from which everyone in the class may benefit hearing the answer; in those circumstances I may copy in the class email list when I reply.

Netiquette

Students are expected to abide by professional standards in all written and spoken communications, including email, web-based and other electronic communications. I will not respond to emails without a subject line or appropriate salutation. For a guide to respectful electronic communications, please see: http://www.albion.com/netiquette/corerules.html

Acknowledgements

In designing this course I looked closely at 1) how other faculty members teach program evaluation at UGA and 2) program evaluation courses at the nation's other top schools of public affairs and public policy. To that end I gratefully acknowledge that some of the ideas, readings, and cases I use in this course have been borrowed from policy analysis courses such as those at the University of Washington's Evans School of Public Policy and Governance and Indiana University's School of Public and Environmental Affairs.

Course Assignments Overview

This course aims to provide advanced training in both the theory and practice of evaluation work. To that end, all of the written work you will produce--either individually or in groups--relates to the development of an actual evaluation proposal.

The final course product--the group evaluation proposal--will be a full proposal designed in accordance with a 2013 US Department of Labor solicitation for the Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant program. Specifically, community colleges seeking TAACCCT grants are required to submit an evaluation plan as part of their grant proposal. Thus, a community college applying for a TAACCCT grant in turn solicits proposals from prospective evaluators--including a description of work and budget, selects the best proposal, and then submits this evaluation plan as part of the grant program application. The case we will specifically consider is Bellingham Technical College's (BTC) Building Nursing Pathways project. Thus, your group is tasked with developing a full evaluation proposal just as if you were seeking to earn an evaluation contract from BTC as part of a TAACCCT grant.

As the semester progresses, I will distribute detailed guidance regarding each stage of the project (e.g., idea sketch, working draft, etc.).

General Thoughts on Assignments

This course aims to provide advanced training in both the theory, practice, and consumption of program evaluation. To that end, coursework consists almost solely of producing deliverables just as one might when working as an analyst or manager at a public, private, or non-profit organization. Accordingly, I expect your assignments to be well organized, thoughtful, clearly written, and free of typos and errors. Please do not turn in any assignment that has not first been edited for grammar and typos both by you and by another individual (a friend, a classmate, your partner, etc.) When I read and grade your assignments I am evaluating your ability to critically analyze a policy issue; typos, poor sentence structure, and grammatical errors distract me from what is important. Your assignments for this class fall in two categories: individual work and group work. Four written assignments (done as group products) capture the thinking of evaluation design team; Several other smaller assignments, done individually, parallel the team's evaluation design work, both feeding it and reflecting on it.

A few notes regarding deliverables:

• All assignments are due to the course @eLearning page before class on the designated week
• Files must be clearly labelled using the following format: 'Firstinitial.Lastname.PADP8680.Assignment#'. Assignments that do not follow this convention will not be accepted.
• All written work must be double spaced, with 1-inch margins.
• All assignments must be submitted in .pdf format. Assignments not submitted as a .pdf file will be returned for revision and will be marked late.
• The two preliminary group memos (Plan and Design Memo and Data and Analysis Memo) must have line numbers to facilitate peer review.
• Memos without line numbers will be returned for revision and will be marked late.
• All group work must have a title page that includes the names of all group members (this page does not count towards page limit).

Deliverables Schedule

Deliverables

Please note that specific details for each assignment will be passed out under separate heading and cover. The text below is simply mean to provide an overview of how the course assignments fit together. As the semester progresses, I will distribute detailed guidance regarding each stage of the project (e.g., idea sketch, working draft, etc.). Those documents will also be posted on this page.

1. Initial Idea Sketch

This individual assignment asks you to bring to the first meeting of your evaluation design team, your 'read' of the RFP in terms of what is it mainly (and really) asking for, what do you know about the topic, what ideas do you have about how an evaluation can address it.

2. Plan and Design Memo

This group memo will offer preliminary ideas about the policy evaluation problem, how it may be approached, and the relevant conditions and constraints in the context for evaluation. The Memo will describe the cast of characters, assumptions about the design problem in context, evaluation questions and purposes, and possible approaches. For reference, this memo should primarily cover items found in sections II (Introduction), III (Intervention), and IV (Implementation Analysis) of the Detailed evaluation template

3. Peer Review: Plan and Design Memo

For this individual assignment you will review another team's design plan and provide feedback based upon our course discussion and readings (The exercise will also serve to gather ideas for improving your own design).

4. Data and Impact Analysis Memo

This group memo will offer your group's preliminary plan for the data procurement and analysis components of your proposal. In other words, how will you actually measure and assess program impacts? Your memo should describe the your data collection plan, measurement strategy(ies), and quantitative and/or qualitative analysis techniques. This memo should primarily cover items found in section V (Outcomes/ Impact Analysis Design) of the Detailed evaluation template

5. Peer Review of Impact Analysis Memo

For this individual assignment you will review another team's impact analysis plan and provide feedback based upon our course discussion and readings (The exercise will also serve to gather ideas for improving your own design).

6. Final Presentation

Your group will give a brief (5-10 minute) presentation of your evaluation during the last week of class.

7. Final Proposal

This group assignment will be a review-worthy draft of your team's overall evaluation research strategy that responds to the RFP, outlines and describes the design components, sets out a timeline, notes resource requirements (as best you can tell) and relevant political considerations in the process of developing and implementing the evaluation. This deliverable should in large part consistent of your evolving Preliminary Design Sketch and Impact Analysis Memo. In other words, you will weave together these two pieces, account for peer review comments, and add additional components as necessary to produce the overall plan.

All required readings are linked to in the following calendar or provided on the course website (password protected). Many are also available on Google Scholar or from the UGA Library. Please note that I have purposely assigned only 2-4 readings per week, with the strong expectation that this will allow you to read each piece completely and fully engage with the readings. While this course is not designed with tests or quizzes, if it becomes apparent that students are not doing the reading I reserve the right to give in-class quizzes on the material.

Optional Texts

While no text is required for the course, you may wish it purchase one or more of these books for easy reference and future use:

• Wholey, Joseph S., Harry P. Hatry, and Kathryn E. Newcomer. Handbook of Practical Program Evaluation. Vol. 19. John Wiley & Sons, 2010.
• Rossi, Peter H., Mark W. Lipsey, and Howard E. Freeman. Evaluation: A systematic approach. Sage Publications, 2003.
• Murnane, Richard J., and John B. Willett. Methods Matter: Improving causal inference in educational and social science research. Oxford University Press, 2010.
• Angrist, Joshua D., and Jorn-Steffen Pischke. Mostly Harmless Econometrics: An empiricist's companion. Princeton University Oress, 2008.

Reading Schedule