EXECUTIVE OFFICE OF THE PRESIDENT OFFICE OF MANAGEMENT AND BUDGET WASHINGTON, D.C. 20503
March 10, 2020 M-20-12 MEMORANDUM FOR HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES FROM: Rus_sell T_. VoughtQ � J__ Actmg D1rector �
SUBJECT:
Phase 4 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Program Evaluation Standards and Practices
1) Introduction
This Memorandum provides guidance to CFO Act agencies necessary to meet certain requirements of the Foundations for Evidence-Based Policymaking Act of 2018 ("Evidence Act"),1 which was signed into law on January 14, 2019. The Evidence Act advances program evaluation as an essential component of Federal evidence building. 2 Evaluation is an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency. This Memorandum provides program evaluation3 standards to guide agencies in developing and implementing evaluation activities, evaluation policies, and in hiring and retaining qualified staff. It also provides examples of leading practices for agencies to draw upon as they build evaluation capacity, develop policies and procedures, and carry out evaluations to support evidence-based policymaking. Future guidance will address how agencies should use the information generated from evaluations and other evidence-building activities to more effectively deliver on mission.
While this guidance applies by law to CFO Act agencies, all agencies are strongly encouraged to implement the standards and best practices discussed in this Memorandum. This Memorandum describes the high-level consensus standards and practices currently recognized for their value in supporting a variety of Federal evaluation needs. These standards are: relevance and utility, rigor, independence and objectivity, transparency, and ethics. The practices described herein were selected for their potential high usefulness in supporting agencies' implementation of the standards. The standards in this guidance are designed to improve the quality and use of evaluation across Federal agencies, while recognizing that agencies must build policies, evaluation offices, and infrastructure that meet their distinct evaluation needs and responsibilities. These standards and practices are intended to inform both 1 Pub. L. No. 115-435, 132 Stat. 5529 (2019), available at https://www.congress.gov/J l5/plaws/publ435/PLA W-l l 5publ435.pdf.
2 See Office of Mgmt.
and Budget, Exec. Office of the President, 0MB M-19-23, Phase I Implementation of the Foundations/or Evidence-Based Policymaking Act o/2018: Learning Agendas, Personnel, and Planning Guidance (2019), available at https://www.whitehouse.gov/wp-content/uploads/2019/07/M-l 9-23.pdf. See specifically Appendix A: Components of Evidence.
3 For the purposes of this guidance, "program evaluation" and "evaluation" are synonymous. The term "evaluation" signifies "an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency." 5 U.S. C. § 311 (3 ). "Evaluation" and "program" are further described in Appendix A.
specific evaluations and the broader set of evaluation activities. As such, these standards and practices apply not just to Federal evaluation offices, but also have applicability to other Federal units that carry out or sponsor evaluation and to individual evaluators, including Federal evaluation staff, outside partners, and recipients of Federal awards that are performing work on behalf of the agency.
This guidance reflects current thinking on evaluation standards and practices for Federal agencies and may be updated to address changes in the evaluation landscape and to incorporate new practices developed over time. Federal agencies are expected to consider current context as they apply these standards and practices (e.g., evaluation purpose or timing or agencies' resources) and the need for continuous improvement in the application of standards and practices (e.g., agency evaluation capacity should mature over time).
2) Background
The Evidence Act states that "the Director of the Office of Management and Budget, in consultation with any interagency council relating to evaluation, shall-(A) issue guidance for program evaluation for agencies consistent with widely accepted standards for evaluation; and
(B) identify best practices for evaluation that would improve Federal program evaluation."4 This guidance addresses the law's requirement, which 0MB has categorized as part of Phase 4 of implementing the Foundations for Evidence-Based Policymaking Act.5 As stated in 0MB M-19-23, meeting the requirements of the Evidence Act "will necessarily build on activities already underway and, in some cases, require coordination of these activities in new ways."6 0MB convened an interagency council to consult on the development ofthis guidance and used a process that reflected this coordinated approach.
In the course of the consultation, the interagency council reviewed an extensive list of source documents to identify widely accepted standards for evaluation. These sources include, but are not limited to: Federally-commissioned reports by the National Academies, such as Principles and Practices for Statistical Agencies and Principles and Practices for Federal Program Evaluation; publications by the Commission for Evidence-Based Policymaking; Monitoring and Evaluation Guidelines for Federal Departments and Agencies that Administer United States Foreign Assistance (0MB M-18-04); 0MB Statistical Policy Directives; Analytical Perspectives of the Budget FY13-FY20; 2018 Reform Plan and Reorganization Recommendations; guidelines and principles published by the American Evaluation Association; a wide cross-section of Federal agency evaluation policies and guidelines; reports and guidance issued by the U.S.
Government Accountability Office; and standards and guidelines issued by international agencies and non-governmental organizations. The Bibliography in Appendix D includes sources that directly influenced the standards or practices. 4 Evidence Act§ lOl(e)(l).
5 See 0MB M-19-23, at 3.
6 Id.
2
Additionally, 0MB convened listening sessions with evaluation experts outside the Federal government, a cross-section of Federal evaluation and statistical experts, and the newly designated Federal Evaluation Officers to obtain input on the content and implementation of the standards and practices. The interagency council also consulted additional stakeholders in 0MB and at various Federal agencies.
To be applicable across the wide range of Federal Agencies, the interagency council advised that the standards be articulated as principles to give Federal agencies a framework for building and maintaining high-quality evaluation capacity, while allowing for appropriate flexibility in meeting Evidence Act requirements and executing other evaluation activities. These standards apply to all Federal evaluation activities, though agency capacity varies. Each agency should be moving toward fully implementing the standards as it builds and enhances its capacity.
Similarly, each agency must determine the applicability of the practices given its unique evaluation capacity and the needs of each evaluation activity. As briefly described in 0MB M-19-23, these standards for evaluation are designed to operate in tandem with key Federal statutes,7 as well as implementing guidance for these statutes issued by 0MB. Additionally, these standards and practices recognize the implementation of evaluation and evidence-building activities of the Evidence Act as a complement to principles of the Paperwork Reduction Act of 1995, OMB's implementing guidance for the Information Quality Act, and the Performance Framework8 outlined in the Government Performance and Results Act Modernization Act of 2010.9 These standards and practices focus on evaluation, as is mandated by the Evidence Act, and are not addressed by other Federal policies and frameworks.
3) Summary of Federal Program Evaluation Standards
Evaluators need to practice and embody these standards in their work in order for Federal evaluations to have the credibility needed for full acceptance and use. Each evaluation standard requires the integration of all of the other standards; however, at times adherence to one or more of these standards must be judiciously balanced with adherence to others. Appendix B includes a more detailed explanation of the standards.
7 For example, the Information Quality Act, Privacy Act, and Paperwork Reduction Act of 1995.
See Office of Mgmt. & Budget, Exec. Office of the President, 0MB M-19-15, Improving Implementation of the Information Quality Act (2019), available at https://www.whitehouse.gov/wp-content/uploads/20 l 9/04/M-19-15 .pdf.
8 See Office of Mgmt.
and Budget, Exec. Office of the President, 0MB Circular No. A-11, Preparation, Submission and Execution of the Budget §§ 200, 210, 230, 240, 260, 290 (Dec. 2019), available athttps://www.whitehouse.gov/wp content/uploads/2018/06/al l .pdf. 9 GPRA Modernization Act of2010 ("GPRA" or "GPRA Mod"), Pub. L. No. 111-352, 124 Stat. 3866 (2011), available at https://www.govinfo.gov/content/pkg/PLA W-l l lpubl352/pdf/PLA W-111 publ352.pdf.
3
RELEVANCE AND UTILITY
Federal evaluations must address questions of importance and serve the information needs of stakeholders in order to be useful resources. Evaluations should present findings 10 that are actionable and available in time for use. Information should be presented in ways that are understandable and that can inform agency activities and actions such as budgeting, program improvement, accountability, management, regulatory action, and policy development.
RIGOR Federal evaluations must produce findings that Federal agencies and their stakeholders can confidently rely upon, while providing clear explanations of limitations. The quality of an evaluation depends on the underlying design and methods, 11 implementation, and how findings are interpreted and reported. Credible evaluations must be managed by qualified evaluators with relevant education, skills, and experience for the methods undertaken. An evaluation must have the most appropriate design and methods to answer key questions, while balancing its goals, scale, timeline, feasibility, and available resources.
INDEPENDENCE AND OBJECTIVITY
Federal evaluations must be viewed as objective in order for stakeholders, experts, and the public to accept their findings. This depends on the independence and objectivity of the evaluators. Federal agencies should enable evaluators to, and evaluators should, operate with an appropriate level of independence from programmatic, regulatory, policymaking, and stakeholder activities. While stakeholders have an important role in identifying evaluation priorities, the implementation of evaluation activities, including how evaluators are selected and operate, should be appropriately insulated from political and other undue influences that may affect their objectivity, impartiality, and professional judgement. Evaluators should strive for objectivity in the planning and conduct of evaluations and in the interpretation and dissemination of findings, avoiding conflicts of interest, bias, and other partiality.
TRANSPARENCY Federal evaluation must be transparent in the planning, implementation, and reporting phases to enable accountability and help ensure that aspects of an evaluation are not tailored to generate specific findings. Decisions about the evaluation's purpose and objectives (including internal versus public use), the range of stakeholders who will have access to details of the work and findings, the design and methods, and the timeline and strategy for releasing findings should be clearly documented before conducting the evaluation. These decisions should take into consideration any legal, ethical, national security, or other constraints for disclosing information.
Once evaluations are complete, comprehensive reporting of the findings should be released in a timely manner and provide sufficient detail so that others can review, interpret, or replicate/reproduce the work. 10 In this guidance, "findings" refers to results, conclusions, and recommendations that are systematically generated through analyzing and interpreting data. See Appendix A: Definitions.
11 In this guidance, the term "design and methods" is used to collectively address the structure of an evaluation, inclusive of evaluation approach; variables for, conditions under, timing of, and sources from which data are used or collected; and quantitative and qualitative data collection and analysis methods. See Appendix A: Definitions.
4
ETHICS Federal evaluations must be conducted to the highest ethical standards to protect the public and maintain public trust in the government's efforts. Evaluations should be planned and implemented to safeguard the dignity, rights, safety, and privacy of participants and other stakeholders and affected entities. Evaluators should abide by current professional standards pertaining to treatment of participants. Evaluations should be equitable, fair, and just, and should take into account cultural and contextual factors that could influence the findings or their use.
4) Summary of Evaluation Practices
This guidance includes examples ofleading evaluation practices, listed below, that can support the evaluation standards. Appendix C includes a more detailed explanation of these practices. As agencies develop multi-year Learning Agendas, establish Annual Evaluation Plans, and conduct other evaluation activities, these practices may be helpful for the Evaluation Officer and Federal evaluation staff in their roles as champions of evidence and evaluation in their agencies. During deliberation on these practices, the interagency council and other stakeholders highlighted the practical reality that what works best and under what conditions will continue to evolve as the evaluation field advances and as evaluation functions in an agency mature.
The set of practices and the descriptions in Appendix C are not meant to be exhaustive of the efforts an agency, office, or program could, should, or must undertake to ensure the quality and integrity of evaluation or adherence to legal or other requirements. The practices aim to provide greater specificity and detail on what may be useful when planning and implementing evaluation activities to fulfill the goals of the standards. These practices are not intended to be a substitute for the extensive literature on evaluation theory, methods, and operations, and the application of these and other evaluation practices requires evaluation expertise and judgment that also balances usefulness, need, and efficient allocation of resources. Agencies may be starting at different places when using these practices to carry out evaluation activities. Consequently, agencies should consider their prior evaluation experience, existing expertise, and current needs before engaging in specific activities or strategies.
1. Build and Maintain Evaluation Capacity:
Staff the Federal evaluation workforce with qualified personnel and support their continued professional development in order to effectively plan, manage, implement, and oversee high-quality evaluation activities.
2. Use Expert Consultation Effectively:
Expand the content knowledge and technical expertise of the evaluators designing and conducting an evaluation by allowing for supplemental expert input and review at critical junctures.
3. Establish, Implement, and Widely Disseminate an Agency Evaluation Policy:
Ensure that evaluation activities adhere to an evaluation policy and that stakeholders are aware of its content and use. ,
4. Pre-Specify Evaluation Design and Methods:
Make an evaluation's design and methods available before the evaluation is conducted and in sufficient detail to achieve rigor, transparency, and credibility by reducing risks associated with the adoption of inappropriate methods or selective reporting of findings, and instead promoting accountability for reporting methods and findings.
5
5. Engage Key Stakeholders Meaningfully:
Identify and involve critical internal and external stakeholders to help ensure relevant and useful evaluation activities.
6. Plan Dissemination Strategically:
Ensure key audiences are aware of and understand evaluation activities broadly and each evaluation's purpose, progress, and findings.
7. Take Steps to Ensure Ethical Treatment of Participants:
Put measures in place to ensure the dignity, rights, safety, and privacy of participants, other individuals, and entities affected by an evaluation.
8. Foster and Steward Data Management for Evaluation:
Establish and maintain an organizational culture that values protecting the integrity, security, privacy, and confidentiality of data when carrying out evaluation activities.
9. Make Evaluation Data Available for Secondary Use:
Establish procedures that facilitate and promote reuse of data from evaluations, taking into consideration any legal, ethical, or other constraints for disclosing the data.
10. Establish and Uphold Policies and Procedures to Protect Independence and
Objectivity: Ensure that Federal agencies establish and maintain the necessary policies and procedures to ensure that evaluation offices and staff have the authority to approve an evaluation's design and methods and release evaluation findings to safeguard against bias.
5) Implementation
The standards and practices referenced in this guidance should be used to improve Federal program evaluation and support Evaluation Officers and other staff as they implement Evidence Act requirements as outlined by law and other 0MB guidance. 12 These requirements include, but are not limited to: the agency evaluation policy; Leaming Agendas (evidence-building plans); Annual Evaluation Plans; and Capacity Assessments. The standards and practices should also inform other evaluation activities as they are planned and implemented. Implementation of the standards and practices includes sub-agencies, operational divisions, and bureaus within CFO Act agencies; non-CFO Act agencies are also strongly encouraged to implement the standards and practices to support their evaluation and evidence-based policymaking activities.
Except where otherwise indicated, this guidance will be implemented by (1) Evaluation Officers, who are expected to play a leading role in overseeing the agency's evaluation activities and Leaming Agenda, as well as collaborating with, shaping, and making contributions to other evidence-building functions within the agency; 13 and (2) agency evaluators and staff in related functions who support the development and use of evaluation, using technical expertise and knowledge of evaluation methodology and these standards for evaluation and related analytic activities within Federal agencies.
12 See, e.g., GPRA Mod,§§ 306 (a)(8), 1115(b)(I0), 1116 (c)(3)(C) (explicitly encouraging evaluation and use of evidence); U.S. Gov't Accountability Office , GAO-11-6 46 SP, Performance Measurement and Evaluation: Definitions and Relationships (May 2011 ), available at https://www.gao.gov/assets/80/77277.pdf ( describing and explaining "the relationship between ...
performance measures and program evaluation").
13 See Office of Mgmt.
& Budget, Exec. Office of the President, 0MB Circular No. A-11, § 290.4 (Dec. 18, 2019) available at https://www.whitehouse.gov/wp-content/u ploads/2018/06/al 1.pdf#pa ge=762. 6
This guidance should be used to continually improve the capacity of Federal agencies to generate evidence about effectiveness and implementation, identify areas for improvement of programs, policies, or organizations, and inform mission-critical decisions and policies. Future guidance will articulate specific requirements for how the evidence generated from evaluations and other activities should be used in agency budget justifications, strategic plans, and other processes. Recognizing the wide range of Federal evaluation activities, this guidance is designed to apply to evaluations that may range in importance and scope, and agencies must apply these standards flexibly in a common sense and workable manner. 0MB sought to avoid detailed, prescriptive, "one-size-fits-all" guidelines that would require different types of evaluation activities to be treated in the same manner. While these standards and practices will assist in establishing a more formal structure for Federal evaluation, they should not be used to introduce administrative rigidity and complexity, which may detract from innovation in developing and maintaining agencies' evaluation capacity'. The standards and practices should be implemented with recognition of the distinct circumstances and capacities of each agency. Implementation should leverage the knowledge of stakeholders and evaluation experts both internal and external to the Federal government.
7
Appendix A: Definitions
Agencies should use these definitions when implementing the standards and practices in this guidance, Evidence Act requirements related to evaluation, and other evaluation activities. These definitions may be superseded by future laws, 0MB Circular No. A-11, or other 0MB guidance, in which case agencies should be guided by those updates.
Descriptive Studies can be quantitative or qualitative in nature, and seek to describe a program, policy, organization, or population without inferring causality or measuring effectiveness. While not all descriptive studies are evaluations, some may be used for various evaluation purposes, such as to understand relationships between program activities and participant outcomes, measure relationships between policies and particular outcomes, describe program participants or components, and identify trends or patterns in data.14 Design and Methods are used in this memo to collectively address the structure of an evaluation that answers the evaluation question(s), inclusive of evaluation approach; variables for, conditions under, timing of, and sources from which data are used or collected; and quantitative and qualitative data collection and analysis methods.
Evaluation means "an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency."15 Evaluation can look beyond the program, policy, or organizational level to include assessment of particular projects or interventions within a program, for example, or particular aspects of a policy or functions or units within an organization. Importantly, there are different types of evaluation, each of which address different questions (see Formative Evaluation, Impact Evaluation, and Process/Implementation Evaluation). Evaluations may address questions related to the implementation or institution of a program, policy, or organization; the effectiveness or impact of specific strategies related to or used by a program, policy, or organization; and/or factors that relate to variability in the effectiveness of a program, policy, or organization or strategies of these. Evaluations can also examine questions related to understanding the contextual factors surrounding a program, as well as how to effectively target specific populations or groups for a particular intervention. They can provide critical information to inform decisions about current and future programming, policies, and organizational operations.
Finally, evaluations can and should be used for learning and improvement purposes, as well as accountability purposes. 16 Evaluation Activities include the planning, implementation, management, and reporting of activities overseen or coordinated by evaluators and related staff within a Federal agency. This includes, but is not limited to: developing and coordinating multi-year LearningAgendas, establishing Annual Evaluation Plans, planning and managing or conducting specific evaluations, summarizing evaluation findings for particular programs or policies, supporting
14 See 0MB Circular No.
A-11, § 200.22, at 641, available at https://www.whitehouse.gov/wp content/uploads/2018/06/a11.pdf#page=641 (describing the term "descriptive studies").
15 Evidence Act§ 101(e)(4)(B) (citing 5 U.S.C.
§ 311(3)).
16 See 0MB Circular No.
A-11, § 200.22, at 640 available athttps:l/www.whitehouse.gov/wp content/uploads/2018/06/al 1.pdf#page=640 (see first paragraph under the definition of "evaluation"). 8
other offices within an agency to interpret evaluation findings, and bringing evaluation-related evidence to bear in decision-making. Evaluators are Federal staff and associated partners who are trained-through advanced education and evaluation experience ( e.g., quantitative, qualitative and/or mixed-method evaluation specializations)-. to properly plan, implement, manage, and/or oversee evaluation activities and evaluations. Some agencies may use interagency agreements, Federal awards, or other agreements to support the planning and implementation of these activities using qualified evaluators.
Evaluation Officers have authority and responsibility for providing leadership over agencies' evaluation and Leaming Agenda activities. As stated in the Evidence Act, the head of each CFO Act agency must designate a senior employee of the agency as the Evaluation Officer of the agency. This shall be done without regard to political affiliation and based on demonstrated expertise in evaluation methodology and practices and appropriate expertise to the disciplines of the agency.17 Non-CFO Act agencies, as well as sub-agencies, operational divisions, and bureaus of CFO Act Agencies are strongly encouraged to designate a qualified Evaluation Officer as appropriate.18 Evidence, as applied in the context of the Federal Performance Framework for improving organizational and agency performance, is viewed broadly as the available body of facts or information indicating whether a belief or proposition is true or valid. As such, evidence can be quantitative or qualitative and may come from a variety of sources, including foundational fact finding ( e.g., aggregate indicators, exploratory studies, descriptive statistics, and other research), performance measurement, policy analysis, and program evaluation. Evidence has varying degrees of credibility, and the strongest evidence generally comes from a portfolio of high quality, credible sources rather than a single source.19 Findings are the principle outcomes of an evaluation; what the evaluation suggested, revealed or indicated. In this guidance, "findings" refer to results, conclusions, and recommendations that are systematically generated through analyzing and interpreting data. The principal parties to an evaluation should ensure that the full evaluation findings with pertinent limitations are made accessible to the persons affected by the evaluation and any others with expressed legal rights to receive the results.20
17 5 U.S.C.
§ 313(b).
18 See 0MB M-19-23, app.
C, at 25-28.
19 See 0MB Circular No.
A-11, § 200.22, at 641, available athttps://www.whitehouse.gov/wp content/uploads/2018/06/al 1.pdf#page=641 (discussing credibility of evidence). Note that evidence is defined in 44 U.S.C. § 3561 ( 6) as "information produced as a result of statistical activities conducted for a statistical purpose," but that 0MB M-19-23 takes a more expansive view of evidence to include foundational fact finding, policy analysis, program evaluation, and performance measurement. See 0MB M-19-23, Appendix A.
20See, e.g., GAO-18-568O, Government Auditing Standards 2018 Revision, ,i,i 6.57, .61, . 70 (2018), available at https://www.gao.gov/assets/700/693136.pdf; U.S. Dep't of Health & Human Servs., Ctrs. for Disease Control & Prevention, Framework/or Program Evaluation in Public Health, Morbidity and Mortality Weekly Report 48 (1999) (No. RR-11) https://www.cdc.gov/mmwr/preview/mmwrhtml/rr481 1 al .htm (Step 6: Ensuring Use and Sharing Lessons Learned).
9
Formative Evaluation is typically conducted to assess whether a program, policy, or organizational approach-or some aspect of these-is feasible, appropriate, and acceptable before it is fully implemented. It may include process and/or outcome measures. However, unlike outcome and impact evaluations, which seek to answer whether the program, policy, or organization met its intended goals or had the intended impacts, a formative evaluation focuses on learning and improvement and does not aim to answer questions of overall effectiveness.21 Impact Evaluation assesses the causal impact of a program, policy, or organization, or aspect thereof, on outcomes relative to those of a counterfactual. In other words, this type of evaluation estimates and compares outcomes with and without the program, policy, or organization, or aspect thereof. Impact evaluations include both experimental22 (i.e., randomized controlled trials) and quasi-experimental designs. An impact evaluation can help answer the question, "does it work, or did the intervention lead to the observed outcomes?"23 Intervention is a combination of program elements or strategies related to the design and implementation of programs and policies designed to produce specific results.
Outcome Evaluation measures the extent to which a program, policy, or organization has achieved its intended outcome(s) and focuses on outputs and outcomes to assess effectiveness. Unlike impact evaluation above, it typically cannot discern causal attribution. Importantly, it is distinct from, but complementary to, performance measurement, as noted below. An outcome evaluation can help answer the question "were the intended outcomes of the program, policy, or organization achieved?" 24 Performance Measurement is the ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals. It is typically conducted by program or agency management. Performance measures may address the type or level of program activities conducted (process), the direct products and services delivered by a program (outputs), or the results of those products and services (outcomes).25 It typically cannot discern causal attribution. Performance measurement is used to measure progress toward goals, and also used to find ways to improve progress, reduce risks, or improve cost-effectiveness. 26
21 See 0MB Circular No.
A-11, § 200.22, at 640--41, available at https://www.whitehouse.gov/wp content/uploads/2018/06/al 1.pdf#page=640.
22 Per U.S.
Gov't Accountability Office, GAO-l 2-208G, Designing Evaluations 40 (rev. 2012), available at https://www.gao.gov/assets/590/588146.pdf, experiments in the evaluation context are most suitable for service and other programs where clearly defined interventions can be standardized and controlled; coverage is limited (small-scale); and randomly assigning participants is feasible and ethical. Experiments measure outcomes, preferably before and after the intervention, for a randomly assigned treatment group and a nonparticipating control group.
23 See 0MB Circular No.
A-I 1, § 200.22, at 640, available at https://www.whitehouse.gov/wp content/uploads/2018/06/a11.pdf#page=640.
24 Id.
25 See U.S.
Gov't Accountability Office, GAO-l 1-646SP, Performance Measurement and Evaluation: Definitions and Relationships (May 2011), available at https://www.gao.gov/assets/80/77277.pdf#page=2.
26 See Office of Mgmt.
and Budget, Exec. Office of the President, 0MB Circular No. A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs 5 (Oct. 1992), available at https :/ /www.whitehouse.gov/sites/whitehouse. gov /files/ omb/circulars/ A94/a094.pdf#page=5.
10
Process or Implementation Evaluation assesses how the program or service is delivered relative to its intended theory of change, and often includes information on content, quantity, quality, and structure of services provided. These evaluations can help answer the question, "was the program, policy, or organization implemented as intended?" or "how is the program, policy, or organization operating in practice?" 27 Program refers to a set of projects or activities28 that support a higher level objective or goal.
For the purpose of this guidance, program includes processes, projects, interventions, policies, operations, activities, entities, and functions. 29 Program operations are the strategies, processes, and activities management uses to convert program inputs into program outputs.30 Program Evaluation. See Evaluation above.
27 See 0MB Circular No .
A-11, § 200.22, at 640, available at https://www.whitehouse.gov /wp content/uploads/2018/06/al I .pdfflpage=640.
28 Per Office of Mgmt.
& Budget, Exec . Office of the President, 0MB M-18-04 , Monitoring and Evaluation Guidelines for Federal Departments and Agencies that Administer United States Foreign Assistance 2 (2018), available at https://www.whitehouse.gov /wp-content/uploads/20 I 7/l l/M-18-04-Final.pdfflpage=3 (defining "program"), note that at some agencies, an activity carries out an intervention or set of interventions through a Federal award, and a project is a set of complementary activities, over an established timeline and budget, intended to achieve a discrete result. An intervention is a combination of program elements or strategies related to the design and implementation of programs and policies designed to produce specific results.
29 See GAO-18-568G , 1 8.08 (describing the term "program"); 0MB M-18-04, at 2 ( defining the term "program"). Consistent with 0MB Circular No. A-11 , this guidance recognizes "that agencies and their stakeholders currently use the term 'program' in different ways. Agencies have widely varying missions and achieve these missions through different programmatic approaches, so differences in the use of the term 'program ' are legitimate and meaningful. For this reason, 0MB does not prescribe a superseding definition of 'program' ; rather, consistent with the GPRA Modernization Act, agencies may identify programs consistent with the manner in which the agency uses programs to interact with key stakeholders and to execute its mission."
0MB Circular No. A-11 , § 200.22, at 647, available at https://www.whitehouse .gov/wp- content/uploads/2018/06/al 1.pdfflpage=64 7. 30 See GAO-18-568G , ,i 8.38(e) (describing the term "program operations"). 11
Appendix B: Federal Program Evaluation Standards RELEVANCE AND UTILITY
Federal evaluations must be relevant, meaning they address important issues within, and sometimes across, agencies. Agencies develop Leaming Agendas and other documents that reflect their long-and short-term goals, link to their Strategic Plan, and reflect their mission.
They identify priority evaluation questions and develop plans for evaluation that contribute valuable evidence and data to inform budget and policy decisions, program design and development, and regulatory actions. The evaluation questions should focus on important agency issues, such as considerations of efficiency, effectiveness, improvement, learning, or accountability. To do so, evaluators build upon the existing scientific literature on the topic, assess the evaluability of policies, programs, and actions under consideration, and design evaluations that reflect the policy, program, and cultural context in which their findings31 will be applied.
To help move an agency's mission forward, relevant evaluations ask and examine important questions and report on them in a way that stakeholders can use the findings. Stakeholder buy-in promotes the use of findings, so throughout the evaluation process, evaluators should consider the cultural context of the programs and participants and involve or consider the stakeholders.
This helps to ensure that evaluation designs and methods32 are suited to the evaluation questions and facilitates use of the findings. Useful evaluations produce findings within a timeline that is appropriate to the questions under consideration, and they use dissemination strategies that enable relevant stakeholders to know of and understand the findings in clear, concise, and actionable ways.
RIGOR Federal evaluations must produce accurate findings that Federal agencies and their stakeholders can confidently rely upon, while providing an understanding of limitations. The quality of an evaluation depends on the planning and implementation of the underlying design and methods, as well as how findings are interpreted and reported. In order to produce credible information and maintain high-quality processes and products, evaluations should be planned, implemented, and interpreted by qualified evaluators with relevant education, skills, and experience for the methods undertaken.
Rigorous evaluation planning must be grounded in a theory of change and take into account existing evidence and gaps in evidence. All evaluations, regardless of method (i.e. qualitative, quantitative, or mixed) must adhere to widely accepted scientific principles and employ methods most appropriate for the evaluation's objectives, within constraints of timeline, feasibility, and available resources. Consistent with the principles of the Paperwork Reduction Act of 1995 and 31 In this guidance, "findings" refer to results, conclusions, and recommendations that are systematically generated through analyzing and interpreting data. See supra Appendix A: Definitions.
32 In this guidance, the term "design and methods" is used to collectively address the structure of an evaluation that answers an evaluation's question(s), inclusive of evaluation type; variables for, conditions under, timing of, and sources from which data are collected; and quantitative and qualitative data collection and analysis methods. See supra Appendix A: Definitions.
12
OMB's implementing guidance for the Information Quality Act, prudent decision making depends on reliable, high-quality information. 33 Once an evaluation approach is determined, the implementation of the design and methods requires methodological rigor. Beyond appropriately implementing the selected methods, evaluators should take actions to strengthen credibility, including soliciting input and feedback through peer review, technical working groups, and/or other consultation from independent, unbiased experts (design and methods credibility); ensuring that inferences about cause and effect are well founded (internal validity); using or collecting verifiable data that accurately capture the intended information and address the questions of interest (measurement reliability and validity); and ensuring clear understanding of policy, program, cultural, and other applicable contexts ( cultural competence) as activities are conducted.
Once data collection and analyses are complete, rigor is necessary in documenting findings and developing reports and other deliverables. Rigorous final products document the populations, settings, or circumstances the evaluation examined and to which findings can be generalized; use precise language to characterize findings accurately based on the design and methods; and disclose limitations, assumptions, methods undertaken, data and sampling frame, and justifications for any changes in design and methods from the initial plan.
INDEPENDENCE AND OBJECTIVITY
Federal evaluations must be independent and objective. These core and complementary principles of evaluation depend on the independence and objectivity of the evaluator(s). Evaluators should be inclusive and engage a broad range of relevant stakeholders in setting evaluation priorities, identifying evaluation questions to be answered, and assessing the implications of findings, rather than depending upon the evaluators alone or on a limited group of voices. However, Federal agencies should enable evaluators to, and evaluators should, operate with an appropriate level of independence from programmatic, regulatory, and policymaking activities. While stakeholders have an important role in identifying evaluation . .
priorities, the implementation of evaluation activities, including how evaluators are selected and operate, should be appropriately insulated from any political and other undue influences that may affect their objectivity, impartiality, and professional judgement. Evaluators should have the necessary authority to protect their independence and objectivity in the design and conduct of evaluations, and in the interpretation and dissemination of findings.
Objective information is impartial, unbiased, clear, and complete. Lack of objectivity compromises the evalt1:ation findings and their uses. Evaluators should have qualifications appropriate to the evaluation and be free from concerns about bias or partiality based on prior work. Evaluators should acknowledge and attempt to mitigate conflicts of interest or bias and other partiality that may be introduced in how they frame evaluation questions, collect or analyze data, or interpret findings. Objectivity is further maximized by making comprehensive information about evaluation design, methods, and findings broadly available on a timely basis.
33 See Office of Mgmt.
& Budget, Exec. Office of the President, 0MB M-19-15, Improving Implementation of the Information Quality Act 1-2 (2019), available at https://www .whitehouse.gov/wp-content/uploads/20 l 9/04/M-19-15.pdf. 13
TRANSPARENCY Federal evaluation must be transparent in the planning, implementation, and completion phases to enable accountability and help ensure that aspects of an evaluation are not tailored to generate specific findings. Federal agencies and evaluators show a commitment to transparency when they determine and document the purpose and goals of all planned and ongoing evaluations before implementation, including whether the evaluation is for internal purposes only or intended to inform the broader public. The purpose and objectives of an evaluation should then inform decisions about the .range of stakeholders who will have access to the work and findings.
Evaluators should have clear timelines and plans for releasing findings to relevant stakeholders before an evaluation is conducted. There should be a commitment to releasing all findings, including favorable, null, and unfavorable findings, in a timely manner to relevant stakeholders.
Depending on the topic of an evaluation or the agencies involved, there may be legal, ethical, or other constraints on disclosing information. For example, national security might dictate that certain details about an evaluation be withheld or redacted, or some studies may be intended for use only by internal agency staff. Ev.aluators should clearly document these decisions and any disclosure limitations at the outset of an evaluation to enable accountability and foster credibility among stakeholders.
Evaluators should develop a detailed plan describing an evaluation's proposed design, methods, and reporting, along with timelines for implementation. This plan should include information about the process for selecting and engaging stakeholders and experts to inform the evaluation work. Any subsequent departures from the plan, including timelines, should be documented and supported by strong rationale. The plan must also disclose sources of funding and any potential conflicts of interest of the evaluators. Evaluators should disclose this information before an evaluation is undertaken to help ensure that aspects of design, methods, and reporting are not tailored, either intentionally or unintentionally, to generate specific findings.
After an evaluation is completed, the implemented design, methods, and findings should be reported in sufficient detail such that readers can determine whether the evaluation meets standards for rigor and independence, assess the credibility and objectivity of the findings, and replicate/reproduce the work. Evaluation reports should include full descriptions of any limitations of the evaluation, including those related to design and methods, implementation, generalizability, or interpretation of findings. Whenever possible, the data collected during an evaluation should be made available to support replication/reproduction and secondary use.
Evaluation data must only be disclosed consistent with applicable laws, regulations, and policies to ensure the proper protection of interests such as the security, privacy, and integrity of the data and participants. ETHICS Federal evaluations must be conducted with the highest ethical standards to protect the public and maintain public trust in the government's efforts. Ethical evaluations are planned and conducted to safeguard the dignity, rights, safety, and privacy of participants and other stakeholders and affected entities. Safeguarding privacy requires that evaluators follow appropriate procedures articulated in law and policy when creating, collecting, using, processing, storing, maintaining, disseminating, disclosing, and disposing of data about individuals and 14
entities, whether the data was provided by participants and evaluated entities, or consists of administrative or other data created or obtained from other sources. Evaluators should abide by current professional standards pertaining to the treatment of evaluation participants (including informed consent, confidentiality, and maximizing benefits for the evaluation while minimizing risks to participants). Evaluators have an ethical obligation to ensure that participants in all aspects of the evaluation process fully understand their rights and any inherent risks.
An ethical commitment to fairness and equity for stakeholders requires the intentional effort of evaluators to produce work that is valid, honest, respectful of stakeholders, and considerate of the general public welfare. This applies to all aspects of evaluation design, including providing information that is accessible, intelligible, and usable to meet the needs of stakeholders. To ensure fair, just, and equitable treatment of all persons and affected entities, evaluators should gain an understanding of the range of perspectives and interests that individuals and groups bring to the evaluation, including those not usually represented. This includes accounting for cultural and contextual factors (e.g., languages spoken, political and social climate, power and privilege, or economic conditions) that could influence the evaluation's findings or their use.
In addition to protecting participants and evaluated entities, evaluators should take into account those affected by evaluation findings, including individuals, communities, and organizations. Ethical evaluators identify and inform stakeholders of the evaluation prior to its start and upon significant modification to the evaluation questions or design and communicate findings to affected individuals and entities upon completion of the evaluation.
15
Appendix C: Evaluation Practices
1. Build and Maintain Evaluation Capacity
Effective program evaluation requires agency personnel with specialized evaluation expertise to: 1) coordinate and manage evaluation activities; 2) establish evaluation plans and implement or oversee planned evaluations; and 3) provide evaluation insights on coordinated, enterprise-wide activities such as multi-year Learning Agendas or approaches to including evaluation in funding announcements. Agencies should support continued professional development of evaluation staff. Some examples of approaches that Federal agencies have found useful include
• Hiring staff with specialized experience in program evaluation and establishing teams with collective expertise in an array of evaluation approaches and translation of findings for non-technical audiences. Evaluation knowledge and expertise may come from advanced degrees in program evaluation or related fields with concentration studies in program evaluation. Related fields may include public administration, public policy, implementation science, statistics, economics, social sciences, or other relevant academic disciplines. Direct experience designing and conducting program evaluation may supplement or substitute for advanced degrees. For example, an agency may need to assemble collective expertise and experience in scientific principles and techniques including, but not limited to: theory of change, evaluation feasibility, method selection, logic model development, measurement, mathematical modeling, qualitative coding, focus group protocol, survey design, sampling methods, power analysis, and statistical analysis.
• Leveraging intra-agency, inter-agency, and/or outside expertise through the use of interagency agreements, Federal awards, or other agreements to supplement and enhance capacity. When using external support for evaluation activities, agencies will still need internal evaluation expertise to effectively plan, implement, manage, and oversee evaluation activities.
• Identifying and leveraging opportunities for expert staff to maintain and continue to build their skills and specializations relevant to evaluation through activities such as membership in professional associations, attending and presenting at professional conferences, and keeping up with advancements in the field through internal and external trainings.
2. Use Expert Consultation Effectively
Expert consultation on applicable evaluation approaches and techniques can strengthen an evaluation's design, implementation, and findings. Consulting with experts, internal or external to the agency, can expand the content knowledge and technical expertise of the evaluators, as well as provide critical review of the evaluation as planned, conducted, and reported. This expert input and review can also increase the likelihood that a wider audience will view the findings as credible. Some examples of approaches that Federal agencies have found useful include
• Selecting expert consultants strategically based on areas of scientific, topical, and technical expertise, with due consideration and transparency regarding independence, objectivity, and conflicts of interest.
• Using program, policy, legal, cultural, or other context experts, as necessary, to support gaps in the evaluation team's expertise in order to strengthen the design, implementation, and interpretation of findings. 16
• Being transparent with experts on responsibilities, timelines, and resource constraints.
• Developing a list of experts and documenting the appropriate steps related to their input early in the evaluation process to ensure that they can be incorporated into the evaluation when necessary.
• Engaging expert consultants intentionally throughout the evaluation lifecycle as part of technical work groups, technical expert groups, or peer reviews. For example: o Consulting with technical work group/expert groups, as appropriate, to obtain input on the scientific and technical merit of the evaluation design, its implementation, planned analyses, and findings.
o Using peer review by qualified experts, as appropriate, to obtain input on scientific and technical merit prior to dissemination of evaluation reports and related products. Consistent with 0MB M-05-03 and 0MB M-19-15, peer reviewers often suggest ways to clarify assumptions, limitations, and findings.
For instance, peer reviews can filter out biases; identify oversights, omissions, and inconsistencies; and encourage authors to more fully acknowledge uncertainties.34
3. Establish, Implement, and Widely Disseminate an Agency Evaluation Policy
The Evidence Actrequires that the Evaluation Officer establish and implement an agency evaluation policy. Once the policy is published, it should guide the agency's evaluation activities throughout the evaluation lifecycle. The Evaluation Officer should ensure that the policy is widely known to stakeholders and implemented with fidelity as the agency conducts evaluations. Some examples of approaches that Federal agencies have found useful include
• Ensuring that the agency evaluation policy incorporates the evaluation standards contained in this memo and any other standards and principles deemed important for the agency.
• Making sure that the evaluation policy is written in plain language, avoids the use of jargon, and is accessible to a range of audiences.
• Publishing the evaluation policy publicly and promoting it internally and externally, including posting it on the agency's website and disseminating it through other mechanisms to relevant stakeholders.
• Providing trainings, seminars, and other opportunities in order to make staff at all levels throughout the agency awar~ of the policy and its role in guiding the agency's evaluation activities.
• Referencing and requiring adherence to the evaluation policy in funding opportunities and Federal awards so that evaluation grants, contracts, and cooperative agreements are conducted according to the standards and principles contained in the policy.
4. Pre-Specify Evaluation Design and Methods
Pre-specifying an evaluation's design and methods before conducting the evaluation has several important benefits related to rigor, transparency, and credibility. First, it reduces the risk that decisions about design and methods will be tailored to achieve specific findings. Second, it enables others to independently critique the design and methods and interpret findings properly
34 See Office of Mgmt.
& Budget, Exec. Office of the President, 0MB M-05-03, Final Information Quality Bulletin/or Peer Review 4 (2004), available at https://www.whitehouse .gov/sites/whitehouse.gov/files/omb /memoranda/2005 /m05- 03.pdf#page=4; Office of Mgmt. & Budget, Exec. Office of the President, 0MB M-19-15, Improving Implementation of the Information Quality Act 4 (2019), available at https://www.whitehouse.gov/wp-content/uploads/20 l 9/04/M-19-15.pdf#page=4.
17
in light of how the evaluation was designed and conducted. Notably, it enables users of the findings to distinguish between planned and unplanned analyses. In general, planned analyses produce stronger evidence than unplanned analyses; the latter are more likely to produce "false positive" results, which appear to be reliable but in fact arise from selective adoption of analytic methods. Third, pre-specification helps ensure that evaluators are accountable for reporting the findings of the evaluation, regardless of whether those findings are favorable, unfavorable, or neutral. Some examples of approaches that Federal agencies have found useful include
• Before conducting an evaluation, documenting how pre-specified design and methods will be shared within any legal, ethical, national security, or other constraints on disclosing information.
• Documenting the planned evaluation design and methods in sufficient detail to support future replication and to effectively mitigate the risk that the design and methods will be tailored to achieve specific findings. Depending on the type of evaluation, this could include specifying: o Core evaluation questions and hypotheses to be tested; o Key outcomes and how they will be measured; o Groups to be studied or compared; o Anticipated sample sizes; o Where, when, and how data will be collected; and o An analysis plan that describes how the data will be prepared and analyzed.
• Publishing the pre-specified evaluation design and methods for public accountability. Posting on an appropriate registry or an agency website are ways to address this. For some evaluations, a Paperwork Reduction Act Information Collection Request, which is made public on reginfo.gov, may be required.
• If legal, ethical, national security, or other constraints on disclosing information prevent public pre-specification, then identifying a means to commit to a specific evaluation design and methods internally. Producing a read-only, date-stamped document of design and methods that is accessible to the evaluators and to agency officials is one approach.
• If there are changes to the pre-specified plans, updating the plans in the registry or on the agency website with a date stamp, and clearly documenting these changes in the final evaluation report of findings and other relevant evaluation products. If changes were made after the evaluation began, identify the changes and the rationales for them. For evaluations with constraints on sharing information publicly, this can still be achieved with a date-stamped internal document.
5. Engage Key Stakeholders Meaningfully
Active, strategic engagement of key stakeholders can strengthen support and interest in evaluation; ensure program, policy, or other contexts are accounted for in the evaluation design; and help yield relevant and useful evaluation findings. Federal agencies and evaluators should identify the key entities and people that have interest in the program, policy, and/or specialized topics addressed in an evaluation. This may include, but is not limited to: mission-relevant Federal leadership and staff (both at the agency conducting the evaluation and at agencies with similar missions); state or local agencies; service providers or practitioners; technical and scientific experts; program recipients or participants; and representatives from the communities, organizations, or other contexts where programs occur or that are affected by the policies. Some examples of approaches that Federal agencies have found useful include 18
• Developing a stakeholder engagement plan early when conducting an evaluation. The stakeholder engagement plan should clearly discuss whom to engage, when, for what purpose, and through what mechanisms to ensure that stakeholders are involved throughout the evaluation lifecycle, and should be updated appropriately.
• Selecting a diverse set of stakeholders, as appropriate, who understand a program or policy' s history, purpose, and practical operation in the field and can help ensure that evaluation design and reports are responsive to the needs and current state of the field.
• Engaging stakeholders at different stages of an evaluation, as appropriate.
• Involving key stakeholders in identifying implications of findings and providing important context to help evaluators understand and meaningfully interpret findings. These activities must be undertaken in a manner that safeguards evaluators' independence and objectivity from undue influence and potential bias.
• Documenting and specifying the stakeholders and their respective responsibilities in written plans and reports to promote transparency.
6. Plan Dissemination Strategically
A dissemination strategy is a systematic way to ensure that an evaluation's design, methods, progress, and findings are shared internally and externally using appropriate and accessible mechanisms. Some examples of approaches that Federal agencies have found useful include
• Ensuring that the time and resources to support dissemination are considered early and are integrated in planning an evaluation. This includes revisiting dissemination plans at key stages of the evaluation and revising them as needed.
• Determining the key audience and stakeholders for dissemination, then planning potential products and delivery mechanisms to make information and findings accessible, understandable, and actionable for those audiences.
• Considering how the method or venue for release of information can be most useful to key audiences.
• Monitoring and documenting use of the evaluation products and associated findings after initial dissemination, when appropriate. This could be as simple as periodic checks on citations of the evaluation and associated data.
• Ensuring dissemination is consistent with agency mechanisms and practices developed in accordance with 0MB guidelines on agency responsibilities for information dissemination under the Information Quality Act and other relevant guidelines as applicable. 35
7. Take Steps to Ensure Ethical Treatment of Participants
There are specific actions that evaluators can take to ensure the dignity, rights, safety, and privacy of participants and other individuals and entities involved in evaluations. The following list is not intended to be comprehensive, but rather to highlight specific approaches that may be important depending on the nature of the evaluation. Evaluators should consult with agency privacy and legal personnel to ensure that any approach they take adequately complies with legal
35 See Office of Mgmt.
& Budget, Exec. Office of the President, Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies, 67 Fed. Reg . 8452 (Feb. 22, 2002), available at https://www .federalregister.gov/documents/2002/02/22/R2-59/guidelines-for-ensuring-and-maximizing-the-guality objectivity-utility-and-integrity-of-information; 0MB M-19-15 .
19
and policy requirements.36 Some examples of approaches that Federal agencies have found useful include
• Making sure that evaluators are up-to-date on relevant trainings and certifications prior to working on an evaluation. This might include human subjects' protection training and certification, or making a determination as to whether the Federal Policy for the Protection of Human Subjects (often referred to as the Common Rule) applies to an evaluation. Evaluators must also complete their agency's mandatory privacy awareness and training program.37
• Ensuring that informed consent forms are complete and clear about the benefits and risks of participating in the evaluation and protecting the disclosure of the participant(s) in the evaluation, where appropriate, to prevent retaliation or harm to the individual or entity providing information.
• Making sure that prospective participants receive any support or clarification they might need in order to understand these benefits and risks. This could include electronic communication, letters, informational sessions, or phone calls.
• Ensuring that participants receive complete and accurate information about how their data will be used, including what might be learned about how a program or policy operates, its effectiveness, and how it can be improved, and any protections that will be applied to prevent disclosure of personally identifiable information (see Practice 8 on data stewardship). Participants should understand not only how their data will be used for the evaluation (primary use), but also how and to whom their data might be made available for secondary use (see Practice 9 on making evaluation data available for secondary use).
• Obtaining Institutional Review Board approval prior to data collection or use (required for many evaluations under the Common Rule) is a comprehensive way to ensure many of the steps in this practice.
8. Foster and Steward Data Management for Evaluation
Effective data management, including for data used in evaluations, requires expertise and coordination among a wide range of agency personnel.38 Therefore, agencies should clarify the responsibilities of evaluators in implementing data management practices that uphold an agency's evaluation policy and with regards to applicable data management laws and policies.39 This will include collaboration with the Statistical Official, Chief Data Officer, Senior Agency Official for Privacy, the Chieflnformation Security Officer, or other relevant officials to
36 Agencies are reminded that the Privacy Act (5 U .S.C .
§ 552a), Paperwork Reduction Act (subchapter I, chapter 35, of title 44, U.S . Code) , Sec. 208 of the E-Government Act of2002 (44 U .S.C. § 3501 , note), and other statutes contain legal requirements that complement , and in some cases affect, ethical considerations concerning the collection, use, and disclosure of information about individuals. Failure to comply with these requirements may result in legal liability for the agency or individual employees.
37 Office of Mgmt.
& Budget, Exec. Office of the President, 0MB Circular No . A-130 , Managing Information as a Strategic Resource, app. I,§ 4(h)(l)---(2), (4)---(7) (July 28, 2016), available at https://www .whitehouse.gov/sites/whitehouse. gov /files/ omb / circulars/ A I 3 0/al 30revised. pdf.
38 Office of Mgmt.
& Budget, E xec. Office of the President, 0MB M-19-23 Phase I Implementation of the Foundations for Evidence-Based Policymaking Act o/2018: Learning Agendas, Personnel, and Planning Guidance, app. C , at 20 (2019), available at https://www .whitehouse.gov/wp-content/uploads/2019 /07 /M-19-23.pdf; Office of Mgmt. & Budget , Exec . Office of the President, 0MB M-19-18 , Federal Data Strategy-A Framework for Consistency (2019), available at https://www .whitehouse.gov/wp-content/uploads/20 I 9/06/M- I 9-I 8.pdf.
39 0MB M-19-18.
20
establish frameworks and mechanisms to ensure that the creation, collection, use, processing, storage, maintenance, dissemination, disclosure, and disposition of data for evaluation is permissible, responsible, and appropriate.40 Some examples of approaches that Federal agencies have found useful include
• Fostering an organizational culture for conducting evaluation that values due diligence and informed consent, even when not required by law. Evaluators should identify, document, and communicate the legal authorities to collect, use, and disclose data, as well as applicable limitations for their ability to maintain confidentiality of participant data.41
• Upholding applicable laws, policies, and protocols for assessing and mitigating the risk of disclosure when evaluation data includes personally identifiable information or other sensitive information.
• Establishing training requirements and requiring evaluators to maintain training credentials for protection of human subjects' data, Federal and agency-specific mandatory privacy awareness and training program(s),42 and other legal, ethical, and evaluation method practices or expectations with regard to data management.
• Ensuring the terms and conditions of each Federal award, interagency agreement, or other agreement pursuant to an evaluation are sufficient to meet applicable law and polices for data management applicable to the nature of the evaluation work. Agencies may find it helpful to create template language for common types of applicable laws and evaluation data collections.
• Coordinating among agency officials to ensure an evaluation complies with all aspects of data management, including Federal requirements for equipment, software, or interconnected systems that are used for the creation, collection, use, processing, storage, maintenance, dissemination, disclosure, and disposition of evaluation data or information (e.g., approved/current Authority to Operate declarations or Privacy Impact Assessments)'. 43
9. Make Evaluation Data Available for Secondary Use
Agencies are required to support secondary information dissemination and processing by third parties, thereby making government information accessible, discoverable, and usable.44 Data generated for a specific evaluation may be relevant for future evaluations or other types of statistical purposes by the same agency, other agencies, the broader evaluation field, or the public at large. Some examples of approaches that Federal agencies have found useful include
• Deciding and documenting how and where the evaluation data will be stored, whether or under what conditions the data (or specific elements within the data) can be made available for secondary use, and which specific steps must be taken to meet conditions
40 See 0MB Circular No.
A-130, app. I. 41 0MB M-19-23 explains that the Statistical Official is responsible overseeing the "development of agency confidentiality standards to appropriately safeguard sensitive information contained in the agency's data assets." 0MB M-19-23, app. C, at 29-- 30 ( detailing responsibilities of Statistical Officials). The Chief Data Officer has "the authority and responsibility for, among other things, data governance and lifecycle data management" in coordination with the agency's data governance body, chaired by this official. Id. at 8 ( discussing agency data governance and leadership).
42 0MB Circular No.
A-130, app. I,§ 4(h)(1)(2), (4)-(7).
43 0MB Circular No.
A-130, at 14.
44 0MB M-19-15, at 5.
21
for secondary use prior to beginning any data collection for an evaluation.45 0MB policy requires agencies to ensure that privacy and confidentiality are fully protected and that data are properly secured so that open data do not disclose personally identifiable information. 46
• Making data available for secondary use while protecting against re-identification. It will sometimes be necessary to create new data dissemination products using disclosure limitation techniques, or providing more permissive tiered access to trusted partners who comply with applicable agency procedures. Agencies must account for the "mosaic effect" of data aggregation, which occurs when the information in an individual dataset, in isolation, may not pose a risk of identifying an individual, but when combined with other available information, could pose such risk.47 The mosaic effect demands a risk based analysis,·often utilizing statistical methods whose parameters can change over time.48
• Enabling reuse of evaluation data. Provide a clear data dictionary or other documentation of the source of the data, the meanings of specific data elements, and any limitations on the completeness and accuracy of the data. 0MB policy emphasizes that, when data are made available to the public, potential users must be provided with sufficient information to understand which agency is responsible for the quality of the data being disseminated, as well as the data's strengths, weaknesses, analytical limitations, security requirements, and processing options.49
• Promoting awareness of the evaluation data available for secondary use and providing support for their use. This might be done, for example, through briefings, conference presentations, online evidence reviews, catalogues of an agency's evaluation products, postings on the agency's website, a comprehensive agency data inventory, the Federal Data Catalogue, Data.gov, or any successor site.5 ° For example, a Federal award might be issued to promote secondary use of evaluation data to address agency priority Learning Agenda questions. The Evaluation Officer should coordinate with relevant officials to host challenges, competitions, events, or other initiatives designed to create additional value from public data assets, including evaluation data when possible.51
10. Establish and Uphold Policies and Procedures to Protect Independence and Objectivity
Agencies should establish policies and procedures that support the ability of Federal evaluators to make informed decisions to safeguard against bias and political and other undue influences throughout the lifecycle of evaluation. While agency leadership and other stakeholders play a key role in identifying evaluation priorities, upholding these policies and procedures supports Federal evaluators in protecting the independence and objectivity of the evaluation design and
45 Id.
at 6. See also 44 U.S.C. § 351 l(a) and associated Evidence Act Phase 2 Guidance,forthcoming, describing agencies' responsibility to develop and maintain a comprehensive data inventory and associated metadata.
46 0MB M-19-15, at 5.
47 Id.
at 5--6. 48 The Federal government maintains various resources for assisting agencies with this analysis. For example, the Federal Committee on Statistical Methodology periodically updates guidance for agencies on applying statistical disclosure limitation methods to publicly available datasets and provides technical assistance on applying these methods.
49 0MB M-19-15, at 5.
50 An Agency's Open Data Ombudsman should be available to the public to engage in using public data assets and encourage collaboration. See Evidence Act Phase 2 Guidance,forthcoming, for more detail on engaging the public. 51 See Evidence Act Phase 2 Guidance,forthcoming, for more detail on engaging the public.
22
methods, as well as the interpretation and timely release of findings. Some examples of approaches that Federal agencies have found useful include
• At a minimum, developing and upholding policies and procedures that enable Federal evaluators with appropriate evaluation training and expertise to make decisions about and approve evaluation design and methods, and release and disseminate evaluation findings within legal, ethical, and resource constraints: Identifying the Federal evaluators who are empowered to protect the independence and • objectivity of an evaluation's design and methods and interpretation and release of findings, and specifying their respective responsibilities in the agency evaluation policy.
For example, the Federal staff charged with these responsibilities may include the Evaluation Officer and/or other senior Federal staff and career officials with extensive experience and expertise in program evaluation. These responsibilities could include, but are not limited to, establishing and executing policies and procedures to protect the independence and objectivity of the evaluation design and methods, and release and dissemination of findings; incorporating relevant statutory authorities; and making decisions and approving the scope and technical aspects of the design and methods.
• Developing and executing dissemination policies and procedures that enable qualified Federal evaluators to approve the content of the evaluation reports and other products, as well as advance the timely release of evaluation findings to the public ( or in some cases a more limited range of stakeholders).
• Developing and executing policies and procedures that maintain a balance between incorporating important stakeholder feedback in evaluation reports and the need to release findings in an objective and timely manner. For example, some Federal agencies have policies and procedures that allow key stakeholders to provide feedback on a given report, and if they believe their feedback was not sufficiently addressed, provide the opportunity to write a statement of difference or support that accompanies the report and is posted publicly.
23
Appendix D: Bibliography
Admin. for Children & Families, U.S. Dep't of Health & Human Servs., Common Framework for Research and Evaluation (2016), available at https://www.acf.hhs.gov/sites/default/files/opre/acf common framework for research and eval uation v02 a.pdf. Admin. for Children & Families, Children's Bureau, U.S. Dep't of Health & Human Servs., A Roadmap for Collaborative and Effective Evaluation in Tribal Communities (2013), available at https://www.acf.hhs.gov/sites/default/files/cb/tribal roadmap.pdf.
Admin. for Children & Families, Office of Planning, Research, & Evaluation, U.S . Dep't of Health & Human Servs., ACF Evaluation Policy (2012), available at: https://www.acf.hhs.gov/opre/resource/acf-evaluation-policy. Admin. for Children & Families, Office of Planning, Research, & Evaluation, U.S. Dep't of Health & Human Servs., Behavioral Interventions to Advance Self-Sufficiency (2016), available at: https://www.acf.hhs.gov/opre/research/project/behavioral-interventions-to-advance-self sufficiency.
Admin. for Children & Families, Office of Planning, Research, & Evaluation, U.S. Dep't of Health & Human Servs., The Program Manager's Guide to Evaluation, 2nd E.d. (2010), available at https :/ /www.acf.hhs.gov/ sites/ default/files/ opre/program managers guide to eval2010. pdf.
Admin. for Cmty . Living, U.S. Dep't of Health & Human Servs., Program Evaluations and Reports: Evaluation Policy (2018), available at: https://acl.gov/programs/program-evaluations and-reports. Am. Evaluation Ass'n, AEA Competencies (2018), available at https://www;eval.org/page/competencies.
Am. Evaluation Ass'n, Guiding Principles for Evaluators (Aug . 2018), available at https://www.eval.org/d/do/4381 . Animal & Plant Health Inspection Serv., U.S. Dep't of Agric., Wildlife Services Directive 4.210, Program Evaluation (Mar. 2004), available at: https://www.aphis.usda.gov/wildlife damage/directives/pdf/4.210.pdf.
Bureau for Policy, Planning & Leaming , U.S. Agency for Int'l Dev., Evaluation Policy (May 2017), available at: https://www.usaid.gov/evaluation/policy. Bureau of Conflict & Stabilization Operations, U.S. Dep't of State, Evaluability Assessment Checklist: Working Draft (n.d.), available at https://www.state.gov/wp content/uploads/2018/12/CSO-Evaluability-Assessment-Checklist-Working-Draft.pdf.
24
Ctrs. for Disease Control & Prevention, Program Evaluation Tip Sheet: Integrating Cultural Competence into Evaluation. (n.d.), available at: https://www.cdc.gov/dhdsp/docs/cultural competence tip sheet.pdf. Ctrs. for Disease Control & Prevention, Step by Step, Evaluating Violence and Injury Prevention Policies, Brief 1: Overview of Policy Intervention. (n.d.), available at .
https:/ /www.cdc.gov/injury/pdfs/policy/Brief%201-a. pdf. Ctrs. for Disease Control & Prevention, Framework for Program Evaluation in Public Health. MMWR 1999; 48 (No. RR-11) available at https://www.cdc.gov/mmwr/PDF/rr/rr481 l.pdf. Cultural Competence in Evaluation Task Force, Am. Evaluation Ass'n, Statement on Cultural Competence in Evaluation (Apr. 2011), available at https://www.eval.org/d/do/154.
Iain Chalmers, Trying to do more Good than Harm in Policy and Practice: The Role of Rigorous, Transparent, Up-to-Date Evaluations, 589 ANNALS AM. ACAD. POL. & Soc. Sci. 22- 40 (2003). The Colo. Trust, The Importance of Culture in Evaluation (2007), available at http://www.communityscience.com/pdfs/CrossCultural Guide.r3. pdf.
Comm'n on Evidence-Based Policymaking, The Promise of Evidence-Based Policymaking: Report of the Commission on Evidence-Based Policymaking (2017), available at https://cep.gov/report/cep-final-report.pdf. Comm'n on Evidence-Based Policymaking, Report of the Commission on Evidence-Based Policymaking, Appendices E-H (2017), available at https://cep.gov/report/cep-appendices-e h.pdf.
Corp. for Nat'l and Cmty. Serv., Office of Research & Evaluation, Evaluation Policy (Mar. 2016), available at https://www.nationalservice.gov/sites/default/files/documents/CNCSEvaluationPolicy.pdf. Dev . Assistance Comm ., Org. for Econ . Co-operation & Dev ., Evaluating Development Co operation: Summary of Key Norms and Standards, Second Edition (June 2010), available at https://www.oecd.org/dac/evaluation/dcdndep/41612905.pdf.
Dev. Assistance Comm ., Org . for Econ. Co-operation & Dev ., DAC Guidelines and Reference Series, Quality Standards for Development Evaluation (2010), available at http://www.oecd.org/dac/evaluation/qualitystandards.pdf. Div. of Behavioral & Soc. Sci., Nat'l Acads . of Sci., Eng' g, & Med., Principles and Practices for Federal Program Evaluation: Proceedings of a Workshop (National Academies Press 2017), available at https://doi.org/10.17226/24831.
25
Div. of Behavioral & Soc. Sci., Nat'l Acads. of Sci., Eng 'g, & Med. , Principles and Practices for a Federal Statistical Agency (6th ed., National Academies Press 2017), available at https://doi.org/10.17226/24810. Div. of Research & Learning in Formal and Informal Settings, Directorate for Educ. & Human Res., Nat'l Sci. Found., User Friendly Handbook for Project Evaluation, Chapter 7: A Guide to Conducting Culturally Responsive Evaluations (2010).
Kevin C. Elliot & David B. Resnik, Making Open Science Work for Science and Society, Environmental Health Perspectives, Vol. 127, No. 7. (2019), available at https://ehp.niehs.nih.gov/doi/1 O. l 289/EHP4808 . Euro stat, European Statistics Code of Practice (201 7), available at https://ec.europa.eu/eurostat/documents/4031688/8971242/KS-02-18-l 42-EN-N.pdf/e7f85f07- 91 db-4312-8118-f729c75878c7.
Evaluation Policy Task Force, Am. Evaluation Ass'n, An Evaluation Roadmap for a More Effective Government (Sept. 2019), available at https://www.eval.org/d/do/4747. Evaluation Policy Task Force, Am. Evaluation Ass 'n, Evaluation, Evaluators, and the American Evaluation Association (Oct. 2018), available at https://www.eval.org/d/do/3997.
Fed. R.R. Admin., U.S . Dep't of Transp., Manual for Research, Development, and Technology Program and Project Evaluations (2016), available at https://rosap.ntl.bts.gov/view/dot/31180. Foundations for Evidence-Based Policymaking Act of 2018, Pub. L. No. 115-435, 132 Stat. 5529 (2019), available at https://www.congress.gov/115/plaws/publ435/PLA W-l 15publ435.pdf.
Sarah Gill et al., Health Promotion Practice, U.S. Dep't of Health & Human Servs., Through a Culturally Competent Lens: Why the Program Evaluation Standards Matter, Health Promotion Prac., (Dec. 2016), available at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5137503/pdf/nihms832068.pdf.
Government of Canada, Directive on Results, https://www.tbs-sct.gc.ca/pol/doc eng.aspx?id=31306 (last modified July 1, 2016). Her Majesty's Treasury, United Kingdom, The Magenta Book: Guidance for Evaluation, Part A (2011), available at https ://assets.publishing. service. gov. uk/ government/uploads/ system/uploads/attachment dat a/file/220543/magenta book part a.pdf.
Inst. of Educ. Sci., U.S. Dep't of Educ., What Are WWC Intervention Reports (n.d.), available at https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/What Are WWC Intervention Reports 508.pdf. 26
Inst. of Educ. Sci., U.S. Dep't of Educ., What Works Clearinghouse Procedures Handbook Version 4. 0 (Oct. 2017), available at https :/ /ies.ed. gov /ncee/wwc/Docs/referenceresources/wwc procedures handbook v4. pdf. · Inst.of Educ. Sci., U.S. Dep't of Educ.; Office of the Assistant Sec'y for Planning & Evaluation, U.S. Dep't of Health & Human Servs.; and Chief Evaluation Office, U.S. Dep't of Labor, Exploring a Cross-Agency Platform for Judging Evidence: Resources for Federal Agencies, (Feb. 2013 ), available at https:/ /youth.gov/feature-article/exploring-cross-agency-platform judging-evidence-resources-federal-agencies; https:/ /youth.gov/docs/Common%20Evidence%20Framework-Draft 508 3-11-13 .pdf.
Millennium Challenge Corp., Corporate Policy: Policy for Monitoring and Evaluation, https://www.mcc.gov/resources/doc/policy-for-monitoring-and-evaluation (last modified Mar. 15, 2017). Nat'l Acads. of Sci., Eng'g, & Med., Follow-Up Seminar on Principles and Practices for Federal Program Evaluation (June 2019), available at: https://sites.nationalacademies.org/DBASSE/CNSTA T/DBASSE 17 4630 .
Nat'l Sci. Found. and Inst. of Educ. Sci., Dep't of Educ., Common Guidelines for Education Research and Development (2013), available at https://www.nsf.gov/pubs/2013/nsfl 3126/nsfl 3126.pdf. Office of Evaluation Sci., U.S. Gen. Servs. Admin., Evaluation Policy, (n.d.) available at https://oes.gsa.gov/assets/files/evaluationpolicy.pdf.
Office of Mgmt. & Budget, Exec. Office of the President, Analytical Perspectives, Budget of the US. Government, Fiscal Year 2018, Building and Using Evidence to Improve Government Effectiveness (2017), available at: https://www.govinfo.gov/content/pkg/BUDGET-2018- PER/pdf/BUDGET-2018-PER-4-2.pdf.
Office of Mgmt. & Budget, Exec. Office of the President, Analytical Perspectives, Budget of the US. Government, Fiscal Year 2014, Program Evaluation and Data Analysis (2013), available at: https://www.govinfo.gov/content/pkg/BUDGET-2014-PER/pdf/BUDGET-2014-PER.pdf.
Office of Mgmt. & Budget, Exec. Office of the President, Analytical Perspectives, Budget of the US. Government, Fiscal Year 2017, Building the Capacity to Produce and Use Evidence (2016), available at https://www.govinfo.gov/content/pkg/BUDGET-2017-PER/pdf/BUDGET-2017- PER-4-3.pdf.
Office of Mgmt. & Budget, Exec. Office of the President, Analytical Perspectives, Budget of the US. Government, Fiscal Year 2019, Building and Using Evidence to Improve Government Effectiveness (2018), available at: https://www.govinfo.gov/content/pkg/BUDGET-2019- PER/pdf/BUDGET-2019-PER-4-2.pdf.
27
Office of Mgmt. & Budget, Exec. Office of the President, Analytical Perspectives, Budget of the US. Government, Fiscal Year 2020, Building and Using Evidence to Improve Government Effectiveness (2019), available at https://www.govinfo.gov/content/pkg/BUDGET-2020- PER/pdf/BUDGET-2020-PER-3-2.pdf.
Office of Mgmt. & Budget, Exec. Office of the President, 0MB Circular No. A-11, Preparation, Submission and Execution of the Budget (June 2019), available at https :/ /www.whitehouse.gov/wpcontent/uploads/2018/06/ al 1. pdf. Office of Mgmt. & Budget, Exec. Office of the President, Delivering Government Solutions in the 21st Century: Reform Plan and Reorganization Recommendations (2018), available at https://www.whitehouse.gov/wp-content/uploads/2018/06/Government-Reform-and-Reorg Plan.pdf.
Office of Mgmt. & Budget, Exec. Office of the President, Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity oflnformation Disseminated by Federal Agencies, 67 Fed. Reg. 8452 (Feb. 22, 2002), available at https ://www.federalregister.gov/ documents/2002/02/22/R2-5 9 / guidelines-for-ensuring-and maximizing-the-guality-o bj ectivity-utility-and-integrity-of-information.
Office of Mgmt. & Budget, Exec. Office of the President, 0MB M-05-03, Final Information Quality Bulletin for Peer Review (2004), available at https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/2005/m05-03.pdf. Office of Mgmt. & Budget, Exec. Office of the President, 0MB M-13-13, Open Data Policy Managing Information as an Asset (2013), available at https :/ /www.whitehouse.gov/ sites/whitehouse. gov /files/ omb/memoranda/20 I 3/m-13-13. pdf.
Office of Mgmt. & Budget, Exec. Office of the President, 0MB M-13-17, Next Steps in the Evidence and Innovation Act (2013), available at https://www.whitehouse.gov/wp content/uploads/2018/1 0/m-13-17 .pdf. Office of Mgmt. & Budget, Exec. Office of the President, 0MB M-18-04, Monitoringand Evaluation Guidelines for Federal Departments and Agencies that Administer United States Foreign Assistance (2018), available at https://www.whitehouse.gov/wp content/uploads/2017/11/M-18-04-Final.pdf.
Office of Mgmt. & Budget, Exec. Office of the President, 0MB M-19-15, Improving Implementation of the Information Quality Act (2019), available at https://www.whitehouse.gov/wp-content/uploads/2019/04/M-19-15 .pdf. Office of Mgmt. & Budget, Exec. Office of the President, 0MB M-19-18, Federal Data Strategy-A Framework for Consistency (2019), available at https://www.whitehouse.gov/wp content/uploads/2019/06/M-19-18 .pdf.
28
Office of Mgmt. & Budget, Exec. Office of the President, 0MB M-19-23, Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Learning Agendas, Personnel, and Planning Guidance (2019), available at https://www.whitehouse.gov/wp-content/uploads/2019/07 /M-19-23.pdf.
Office of Mgmt. & Budget, Exec. Office of the President, Statistical Policy Directive No. 1: Fundamental Responsibilities of Federal Statistical Agencies and Recognized Statistical Units, 79 Fed. Reg. 71,610 (Dec. 2, 2014), available at https://www.govinfo.gov/content/pkg/FR-2014-12-02/pdf/2014-28326.pdf.
Office of Mgmt. & Budget, Exec. Office of the President, Statistical Policy Directive No. 4: Release and Dissemination of Statistical Products Produced by Federal Statistical Agencies and Recognized Statistical Units, 73 Fed. Reg. 12,622 (Mar. 7, 2008), available at https://www.govinfo.gov/content/pkg/FR-2008-03-07/pdf/E8-4570.pdf.
Office of the Assistant Sec'y for Policy Dev. & Research, U .S. Dep't Hous. & Urban Dev., HUD Program Evaluation Policy Statement (2016), available at: https://www.federalregister.gov/documents/2016/12/06/2016-29215/hud-program-evaluation policy-policy-statement.
Office of the Sec'y, Ctrs. for Disease Control & Prevention, The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research (1979), available at: https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/index.html. Org. for Econ. Co-operation & Dev., Internal Draft: Ensuring the Good Governance of Evidence: Taking stock of standards for policy design, implementation and evaluation (Publication forthcoming in 2020).
Michael Quinn Patton, Utilization-Focused Evaluation (4th ed., Sage Publishing 2008). Michael Quinn Patton, Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use (The Guilford Press 2011). Hallie Preskill & Nathalie Jones, Robert Wood Johnson Found., A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions (2009), available at https://www.rwjf.org/content/dam/farm/toolkits/toolkits/2009/rwjf48595.
United Nations Evaluations Grp., Norms and Standards for Evaluation (2016), available at: https :/lwww.betterevaluation.org/ en/resources/ example/UNEG-evaluation-standards-2016. U.S. Dep't of Labor, Evaluation Policy (2013), available at: https :/ /www.dol.gov/ agencies/ oasp/ eval uation/Eval uationPolicy.
U.S. Dep't of State, Program and Project Design, Monitoring, and Evaluation Policy (2017), available at https://www.state.gov/wp-content/uploads/2018/12/Department-of-State-Program and-Project-Design-Monitoring-and-Evaluation-Policy.pdf. 29
U.S. Gov't Accountability Office, GAO-12-208G, Designing Evaluations 2012 Revision (Jan. 2012), available at https://www.gao.gov/assets/590/588146.pdf. U.S. Gov't Accountability Office, GAO-10-243, Employment and Training Administration: Increased Authority and Accountability Could Improve Research Program (Jan. 2010), available at https://www.gao.gov/assets/310/300451.pdf.
U.S. Gov't Accountability Office, GAO-19-466, Foreign Assistance: Federal Monitoring and Evaluation Guidelines Incorporate Most but Not All Leading Practices (July 2019), available at https://www.gao.gov/assets/710/700675.pdf. U.S. Gov't Accountability Office, GAO-l 1-646SP, Performance Measurement and Evaluation: Definitions and Relationships (May 2011), available at https://www.gao.gov/assets/80/77277 .pdf.
U.S. Gov't Accountability Office, GAO-17-743, Program Evaluation: Annual Agency-Wide Plans Could Enhance Leadership Support for Program Evaluations (Sept. 2017), available at https://www.gao.gov/assets/690/687526.pdf. U.S. Gov't Accountability Office, GAO-15-25, Program Evaluation: Some Agencies Reported that Networking, Hiring, and Involving Program Staff Help Build Capacity (Nov. 2014), available at: https://www.gao.gov/assets/670/666893 .pdf.
U.S . Gov't Accountability Office, GAO-10-30, Program Evaluation: A Variety of Rigorous Methods Can Help Identify Effective Interventions (Nov. 2009), available at: https://www.gao.gov/new.items/dl030.pdf. U.S. Small Bus. Admin., Framework and Guidelines for Program Evaluation at the US Small Business Administration (Mar. 2019), available at: https://www.sba.gov/document/policy guidance--framework-guidelines-program-evaluation-us-small-business-administration.
W.K. Kellogg Found., The Step-by-Step Guide to Evaluation: How to Become Savvy Evaluation Consumers (Nov. 2017), available at http://ww2.wkkf.org/digital/evaluationguide/view.html with permission of the W.K. Kellogg Foundation. Donald B. Yarbrough et al., The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users (3d ed., Corwin Press 2010), available at: https://jcsee.org/program/ (adopted as the program evaluation standards for the Joint Comm. on Standards for Educ. Evaluation).
30