← Back to research report

Document · 2019-07-10

M-19-23 Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Learning Agendas, Personnel, and Planning Guidance

Complete document.

About this text

This text has been formatted for reading on screen. Page links open the original PDF. Older scans may contain errors in words, tables, or reading order.

Document sections
Jump to a PDF page · 37 included

M-19-23

EXECUTIVE OFFICE OF THE PRESIDENT OFFICE OF MANAGEMENT AND BUDGET WASHINGTON, D.C. 20503

July 10, 2019 MEMORANDUM FOR HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES FROM: , Rus_sell T_. Vought ~ ·J _ ~ Actmg Director \J L- '-'-"-V-'

SUBJECT:

Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Leaming Agendas, Personnel, and Planning Guidance

Background

The Foundations for Evidence-Based Policymaking Act of 2018 ("Evidence Act"), 1 signed into law on January 14, 2019, emphasizes collaboration and coordination to advance data and evidence-building functions in the Federal Government by statutorily mandating Federal evidence-building activities, open government data, and confidential information protection and statistical efficiency. Evidence is broadly defined and includes foundational fact finding, performance measurement, policy analysis, and program evaluation (see Appendix A).2 In addition, the Evidence Act mandates a systematic rethinking of government data management to better facilitate access for evidence-building activities and public consumption.

Despite previous efforts and resource commitments, Federal agencies often lack the data and evidence necessary to make critical decisions about program operations, policy, and regulations, and to gain visibility into the impact of resource allocation on achieving program objectives.

Investing in and focusing on the management and use of data and evidence across the Federal Government will enable agencies to shift away from low-value activities toward actions that will support decision makers: linking spending to program outputs, delivering on mission, better managing enterprise risks, and promoting civic engagement and transparency.

The Evidence Act builds on longstanding principles underlying Federal policies and data infrastructure investments supporting information quality, access, protection, and evidence 1 Pub. L. No. 115-435, 132 Stat. 5529. 2 Pursuant to the Evidence Act, "evidence" is defined as "information produced as a result of statistical activities conducted for a statistical purpose." See 44 U .S.C. § 3561(6). The term "statistical activities" means "the collection, compilation, processing, or analysis of data for the purpose of describing or making estimates concerning the whole, or relevant groups or components within, the economy , society, or the natural environment and ... includes the development of methods or resources that support those activities, such as measurement methods , models, statistical classifications, or sampling frames." See id.§ 3561(10). The term "statistical purpose" means "the description, estimation, or analysis of the characteristics of groups, without identifying the individuals or organizations that comprise such groups and ... includes the development, implementation, or maintenance of methods, technical or administrative procedures, or information resources that support [those] purposes." See id. § 3561(12).

building and use. 3 While anchored on these principles, the Evidence Act creates a new paradigm by calling on agencies to significantly rethink how they currently plan and organize evidence­ building, data management, and data access functions to ensure an integrated and direct connection to data and evidence needs. This paradigm requires engagement and cooperation from multiple actors within agencies ( e.g., senior leadership, policy officials, program administrators, performance managers, strategic planners, budget staff, evaluators, analysts, front-line staff, and data professionals) and key external stakeholders. It accelerates, expands, and reinforces the implementation of many activities underway, including the Federal Data Strategy ("Strategy");4 the government-wide reorganization proposals to strengthen Federal evaluation and reorganize the Census Bureau, Bureau of Labor Statistics, and Bureau of Economic Analysis to improve economic statistical data;5 new monitoring and evaluation guidelines for agencies that administer foreign assistance;6 new centralized mission support shared services and standards; 7 and ongoing open data policy related to open data laws like the DATA Act and Geospatial Data Act.8

3 See, e.g., Treasury and General Government Appropriations Act, 2001, Pub .

L. No . 106-554, § 515(a) (2000); Confidential Information Protection and Statistical Efficiency Act of2002 ("CIPSEA "), Pub. L. No. 107-347, § 502(9)(A)); Data.gov, available at https://www.data.gov (first national data catalog, established in 2009); Exec. Order No. 13,642, Making Open and Machine Readable the New Default for Government Information, 78 FR 28111 (May 9, 2013), available at https://www.federalregister.gov/documents/2013/05/ 14/2013-11533/making-open-and-machine-readable-the-new-default-for• government-information; Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity oflnformation Disseminated by Federal Agencies, 67 FR 8452 (Feb. 22, 2002), available at https://www .federalregister.gov/documents/2002/02/22/R2-59/guidelines-for-ensuring-and-maximizing-the-guality-objectivity­ utility-and-integrity-of-informat; Office of Mgm't & Budget, Exec. Office of the President, 0MB M-13-13 , Open Data Policy­ Managing Information as an Asset (2013), available at https://www .whitehouse.gov/sites/whitehouse.gov/files/omb /memoranda/20 I 3/m-13-13.pdf; Office of Mgm't & Budget, Exec.

Office of the President, 0MB M-13-17, Next Steps in the Evidence and Innovation Agenda (2013), available at https://www .whitehouse.gov/sites/whitehouse.gov/files/omb /memoranda/20 I 3/m-13-17.pdf; Office of Mgm 't & Budget, Exec. Office of the President, 0MB M-14-06 , Guidance for Providing and Using Administrative Data for Statistical Purposes (2014), available athttps://www .whitehouse.gov/sites/whitehouse.gov/files/omb /memoranda/2014 /m-14-06.pdf; and Office of Mgm 't & Budget, Exec. Office of the President, Statistical Policy Directive No . I: Fundamental Responsibilities of Federal Statistical Agencies and Recognized Statistical Units 79 FR 71610 (Dec. 2, 2014), available at https://www.govinfo .gov/content/pkg/FR- 2014-12-02/pdf/2014-28326.pdf.

4 See Office ofMgm 't & Budget, Exec.

Office of the President, 0MB M-19-18 , Federal Data Strategy-A Framework for Consistency (2019), available athttps://www.whitehouse.gov/wp-content/uploads/2019/06/M-19-18.pdf.

5 See Office of Mgm 't & Budget, Exec.

Office of the President, Delivering Government Solutions in the 21st Century Reform Plan and Reorganization Recommendations (2018), available at https://www.performance .gov/GovReform/Reform-and-Reorg­ Plan-Final.pdf.

6 See Office ofMgm 't & Budget, Exec.

Office of the President, 0MB M-18-04 , Monitoring and Evaluation Guidelines for Federal Departments and Agencies that Administer United States Foreign Assistance (2018), available at https://www .whitehouse.gov/wp-content/uploads/2017/l l/M-18-04-Final.pdf.

7 See Office ofMgm 't & Budget, Exec.

Office of the President, 0MB M-19-16 , Centralized Mission Support Capabilities for the Federal Government (2019), available at https://www.whitehou se.gov/wp-content/uploads/2019/04/M-19-16.pdf. 8 Digital Accountability and Transparency Act of 2014, Pub . L. No. 113-101, 128 Stat. 1146, available at https://www.congress .gov/l 13/plaws/publl 01/PLA W-1 I 3publ 101.pdf; Geospatial Data Act of 2018, Pub. L. No. 115-254, div.

B , tit. VII, Subtit. F, 132 Stat. 3413, available at https://www .congress.gov/115/bills/hr302/BILLS-I 15hr302enr.pdf. 2

Iterative and Coordinated Implementation Meeting the requirements of the Evidence Act will require ongoing, focused effort. 0MB recognizes that implementation will necessarily build on activities already underway and, in some cases, require coordination of these activities in new ways.9 0MB is committed to: (1) aligning related data and information policy guidance, such as the Strategy and related 0MB guidance, like 0MB Circular A-11, as applicable; and (2) providing iterative and supplemental guidance in four phases (see Figure 1 below):10

• Phase 1: Learning Agendas, Personnel, and Planning

• Phase 2: Open Data Access and Management

• Phase 3: Data Access for Statistical Purposes

• Phase 4: Program Evaluation

9 See, e.g., CIPSEA; Data.gov; Exec.

Order No . 13,642; 0MB M-13-13; 0MB M-13-17; 0MB M-14-06 ; Statistical Policy Directive No. 1; Comrn'n on Evid.-Based Policymaking, The Promise of Evidence-Based Policymaking (2017), available at https://www.cep .gov/cep-final-report.html (bipartisan report which provided the impetus for Pub . L. No . 115-435).

1° Cf Office ofMgm't & Budget, Exec. Office of the President, 0MB Circular No. A-130 , Managing Information as a Strategic Resource, available athttps://www.whitehouse .gov/sites/whitehouse.gov/files/omb /circulars/ A 130/al 30revised.pdf; Office of Mgm 't & Budget, Exec . Office of the President, Management of Federal Information Resources (Dec. 12, 2000) (revising 0MB Circular No . A-130), available athttps://www.whitehouse.gov /wp-content/uploads/2017 /11/Management-of-Federal­ Information-Resources-Dec 12-2000.pdf; Office of Mgm 't & Budget, Exec. Office of the President, 0MB Circular No . A-11 , Preparation, Submission, and Execution of the Budget (June 20 I 9), available athttps://www .whitehouse.gov/wp­ content/uploads/2018/06/al 1.pdf; 0MB M-13-13; and 0MB M-14-06.

3

Phase 1: Learning Agendas, Personnel, & Planning Learning Age ndas Ch ief Data O' icers Evaluation O icers Statistical O fficials Agency Data Go vernance Boa rds Ch ie Data Officer Counc il Evaluation O icer Counc il o lnteragency Counc il on Statistical Policy Agency Evaluation Plans Capac ity Assessments Ope n Data Plans Learning Agendas, Personnel, & Plannin January 14, 2019 Evidence Act signed into law Figure 1: Evidence Act Guidance Phase 2: Open Data Access & Management Make Data Open by De ault Comp rehensive Data Inventory Federal Data Catalogue o Repos itory of oo ls and Best Practices Open Data Access & Management Phase 3: Data Access for Statistical Purposes Make Agency Data Assets Availab le to Statistical Un its Expand Users' Secure Access to Data Assets through Statistical Un its Allow Recogn ition o Ne w Statistical Un its Standardize App lication Process orAccessing Da ta Assets Cod ify Statistical Un it Respons ibil" ies Data Access for Statistical Purposes Phase 4: Proeram Evaluation Program Evaluation Standards and Best Practices Program Eva luation Skills and Co m petencies (with

OP M )

July 13, 2019 Law takes e ect January 14, 2020 One Year Pos - Ena ctment

6 M onths 12 Months

4 Ongoing Implementation & Reporting o Budget Cycles o Information Resource M anag em ent Plans Performance Plans Strategic Plans Federal Da ta Strategy Annu al Action Plans Bienn ial Report o Statistical Program s Annua l Rep ort Regulatory Plann ing Information Collection Review Ongoing Implementation & Reporting 24+ M onths

Phase 1: Learning Agendas, Personnel, and Planning This memorandum includes the key requirements for implementation of Phase 1 that agencies should begin working on immediately (see Table 1 below). Phase 1 implementation includes developing Leaming Agendas, designating and developing roles and responsibilities for Personnel, and undertaking Planning activities.

I. Learning Agendas: The Learning Agendas are, in many ways, the driving force for several of the activities required by and resulting from the Evidence Act. The creation of Learning Agendas requires agencies to identify and set priorities for evidence building, in consultation with various stakeholders.

II. Personnel: The newly-designated positions (Chief Data Officer, Evaluation Officer, and Statistical Official) all play a key role in leading these activities, including addressing the new reporting requirements. Their corresponding interagency councils likewise support these functions and roles. These senior officials will also serve on a newly required Data Governance Body inside their agency, which will set and enforce priorities for managing data as a strategic asset to support the agency in meeting its mission and, importantly, answering the priority questions laid out in the agency Learning Agenda.

III. Planning: The Evidence Act augments ongoing agency strategic planning activities to promote better use and management of data and evidence, consistent with the GPRA Modernization Act11 and 0MB Circular A-11 Part VI and agency-specific management routines, such as through data-driven performance and strategic reviews. Specifically, Learning Agendas, which identify agencies' priority questions, drive related planning activities, including: o Developing annual Evaluation Plans, which will summarize the specific evaluations an agency plans to undertake to address those questions; o Undertaking Capacity Assessments, which, as part of agencies' strategic plans, will help agencies to assess their ability and infrastructure to carry out evidence­ building activities like foundational fact finding, performance measurement, policy analysis, and program evaluation; and o Identifying the data needed to answer those questions.

Lastly, in tandem with Phase 2 guidance, Open Data Plans will inform dissemination improvement priorities as part of agencies' Strategic Information Resources Management Plans ( discussed below). The body of this memorandum summarizes these initial requirements, the corresponding deadlines, reporting formats, and responsible officials. The appendices that follow provide additional details on how agencies should implement these requirements, including how these requirements tie into existing efforts, structures, and processes. Agencies should review and refer

11 GPRA Modernization Act of 2010 , Pub.

L. No. 111-352, available at https://www.congress.gov/111/plaws/publ352/PLA W­ I I I publ352.pdf. 5

to all parts of this guidance, including the appendices, as they implement the Evidence Act's requirements.12 Agencies are expected to resource these activities appropriately. Table 1: Evidence Act - Initial Implementation Requirements Requirement Deadline D e\'elop L earnino A o I · 1 e· . -. .,, .,enc a (E \'ldence-B ·11· , ,II nm g ,\geullas u1 c m g Plan) .

Document Progress in Developing Learning Agenda (in FY 2021 Evidence Template) Submit Interim Learning Agenda (concurrent with FY 2022 Annual Performance Plan) Submit Annotated Outline of Learning Agenda (part of Initial Draft Strategic Plan) Submit Full Draft Learning Agenda (part of Full Draft Strategic Plan) Submit Final Draft Learning Agenda (part of Final Draft Strategic Plan) Publish Final Learning Agenda (part of Final Strategic Plan) Name Members and Charter a Data Governance Body D Name Chief Data Officer Name Evaluation Officer Name Statistical Official Participate in Designated Official Orientation Participate in Interagency Councils Develop Annual Evaluation Plan Document Progress in Developing Evaluation Plan (in FY 2021 Evidence Template) Complete Evaluation Plan for FY 2022 (concurrent with FY 2022 Annual Performance Plan) September 2019* September 2020* May 2021 September 2021 * December 2021 February 2022 Personnel September 30, 2019 July 13, 2019 July 13, 2019 July 13, 2019 September 2019 Ongoing Planning September 2019* September 2020* Agencies: ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ** ** ** ** ** ** ✓ ✓ ** ✓ ✓ ** ** Responsible Official(s) EO EO EO EO EO EO AH AH AH AH

CDO,EO, SO CDO,EO, SO EO EO

12 0MB recognizes that this guidance uses both the term "data" and the term "information," and that using both terms may cause confusion. The Evidence Act introduced the term "data" into the statutory construct created by the Paperwork Reduction Act, codified in subchapter I of chapter 35 of title 44 of the U.S. Code, which generally speaks in terms of"information." The introduction of the term "data" in the Evidence Act did not, however, disturb the existing definitions or responsibilities that involve the term "information." The result is that the law now describes agency responsibilities using both terms. In this guidance, 0MB uses the terms "data" and "information" consistently with the source of the authority for each requirement.

13 See 31 U .S.C.

§ 90l(b). 6

Conduct Capacity Assessment Propose Approach for Capacity Assessment September 2019* ✓ ** EO (in FY 2021 Evidence Template) Submit Interim Capacity Assessment September 2020* ✓ ** EO (concurrent with FY 2022 Annual Performance Plan) Submit Initial Capacity Assessment May 2021 ✓ ** EO (part of Initial Draft Strategic Plan) Submit Draft Capacity Assessment September 2021 * ✓ ** EO (part of Full Draft Strategic Plan) Submit Final Draft Capacity Assessment December 2021 ✓ ** EO (part of Final Draft Strategic Plan) Publish Final Capacity Assessment February 2022 ✓ ** EO (part of Final Strategic Plan) See Forthcoming Phase 2 Guidance * -Deliverable should be submitted at the time the agency submits information to 0MB per the Budget ,cycle ** - Strongly recommended for non-CFO Act agencies, as well as operational divisions, bureaus, and sub-agencies in CFO Act agencies EO = Evaluation Officer I AH = Agency Head I CDO = Chief Data Officer I SO = Statistical Official I. Develop a Learning Agenda (Evidence-Building Plan)14 Each CFO Act agency should immediately start the process of developing and implementing a multi-year learning agenda that coincides with the four-year timeframe defined for agency strategic plans (see Appendix B for additional implementation details). As described further in Appendix B, non-CFO Act agencies, as well as sub-agencies, operational divisions, and bureaus within CFO Act agencies are strongly encouraged to implement a learning agenda as appropriate in coordination with the agency-wide learning agendas. Below are the key activities for agencies to document progress in developing, publishing, and implementing a learning agenda:

• FY 2021 Budget Submission (September 2019): Concurrent with their FY 2021 Budget submissions in September 2019, agencies must document their progress in developing a learning agenda in the Evidence Template that agencies are required to complete per 0MB Circular No.A-11.

• FY 2022 Annual Performance Plan and Budget Submission (September 2020): Concurrent with their FY 2022 Annual Performance Plan Submission in September 2020 (to coincide with the submission of the FY 2022 Budget), agencies must submit an interim learning agenda.15 Agencies are strongly encouraged to put their learning agenda in place earlier and submit to 0MB prior to the September2020 deadline.

• Draft Strategic Plan (May, September, and December 2021): As part of their initial draft Strategic Plan submitted to 0MB in May 2021, agencies must submit an annotated outline of their updated learning agenda, building on the interim learning agenda submitted in September 2020. The full draft learning agenda must be submitted in 14 A learning agenda is equivalent to the agency evidence-building plan required in Section JOI of the Evidence Act. 5 U.S.C.

§312(a). 15 Because this interim learning agenda is being developed during the last year of agency's current strategic plan, the interim learning agenda should be written to address short-term priority questions that can be answered more immediately and long-term questions that are directly linked to the agency's core statutory mission and therefore are unlikely to differ substantively from one strategic planning cycle to the next.

7

September 2021 with the full draft Strategic Plan. The final draft learning agenda must be submitted in December 2021 with the agency's final draft Strategic Plan.

• Final Strategic Plan (February 2022): As part of their final Strategic Plan posted publicly in February 2022, agencies must include their updated, final learning agenda, which shall cover the same period. II. Constitute Data Governance and Leadership

(a) Data Governance Body By September 30, 2019, the head of each agency must establish an agency Data Governance Body, to be chaired by the Chief Data Officer, with participation from relevant senior-level staff in agency business units, data functions, and financial management (see Appendix C). All agencies must include their Data Governance Body in their Strategic Information Resources Management Plan's Governance sections as required by 0MB guidance.16 All agencies must report their Data Governance Body members to 0MB via email at EvidenceAct@omb.eop.gov no later than September 30, 2019.

(b) Designate Key Senior Officials 0MB recognizes that agencies have been working to put key senior officials in place to meet Evidence Act requirements. As previously communicated over the past few months, the deadlines for designating these positions are listed below.

Chief Data Officer - The Chief Data Officer (CDO) shall have authority and responsibility for, among other things, data governance and lifecycle data management. The head of each agency must designate a senior staff person in the agency with the required qualifications (see Appendix C) no later than July 13, 2019.17 Evaluation Officer - The Evaluation Officer shall have authority and responsibility for providing leadership over the agency's evaluation and learning agenda activities. 18 The head of each CFO-Act agency must designate, without regard to political affiliation, a senior staff person in the agency with the required qualifications (see Appendix C) no later than July 13, 2019. Non-CFO Act agencies, as well as sub-agencies, operational divisions, and bureaus of CFO Act Agencies are strongly encouraged to designate an Evaluation Officer as appropriate.

Statistical Official - The Statistical Official shall have authority and responsibility to advise on statistical policy, techniques, and procedures. The head of each CFO-Act agency must designate a senior staff person in the agency with the required qualifications (see Appendix C) no later than July 13, 2019. 19 Non-CFO Act agencies, as well as sub-agencies,

16 See 0MB Circular No.

A-130, § 5.a. 17 44 u.s.c. § 3520.

18 5 U.S.C.

§ 313. 19 The Act requires the Statistical Official to be the head of an existing statistical agency in CFO Act agencies that have a statistical agency. 5 U.S.C. § 314(a). 8

operational divisions, and bureaus of CFO Act Agencies are strongly encouraged to designate a Statistical Official as appropriate. All agencies must submit the names and contact information for their Chief Data Officer, Evaluation Officer, and Statistical Official to EvidenceAct@omb.eop.gov and post the names on their https://[agency].gov/data webpage no later than August 2, 2019. Agencies must provide any updates to these designations to 0MB via email at EvidenceAct@omb .eop.gov and update their webpages accordingly.

Evidence Act Orientation for Designated Officials - All designated officials must attend an Evidence Act orientation hosted by 0MB in September 2019 to gain a fuller understanding of the requirements under this guidance, develop a common baseline of knowledge about key competencies required for their roles, learn about resources for fulfilling statutory responsibilities, and meet and build relationships with their peers and with 0MB officials to promote interagency coordination and collaboration.

(c) Interagency Councils Designated key officials must also participate with their peers on the following interagency councils: Chief Data Officer Council This Memorandum hereby establishes a Chief Data Officer Council (hereinafter "CDO Council") that will include all agency Chief Data Officers, the Administrator of the Office of Electronic Government ( or designee ), the Administrator of the Office of Information and Regulatory Affairs ( or designee ), and an Ex Officio Member (to represent all Chief Information Officers and Evaluation Officers). 0MB will designate a Chair and constitute a Charter for the CDO Council. The Chair of the CDO Council, in consultation with the GSA Office of Executive Councils, will develop an implementation plan for the Council that will consider the below objectives, as well as lessons learned and best practices for managing Federal-wide executive councils.

The CDO Council shall meet regularly to establish government-wide best practices for the use, protection, dissemination, and generation of data; promote and encourage data sharing agreements between agencies; identify ways in which agencies can improve upon the production of evidence for use in policymaking; consult with the public and engage with private users of Government data and other stakeholders on how to improve access to data assets of the Federal Government; and identify and evaluate new technology solutions for improving the collection and use of data. The CDO Council shares responsibilities with other government-wide councils that conduct statutory data-related activities. 0MB expects that the activities of all these councils will be coordinated through 0MB in order to assure that these activities are complementary and carried out efficiently and effectively.

9

Interagency Council on Statistical Policy The statutory purpose of the Interagency Council on Statistical Policy (ICSP) is to advise and assist the Chief Statistician of the United States in the coordination of the Federal Statistical System; the implementation of statistical policies, principles, standards, and guidelines; and the evaluation of statistical program performance. 20 Its membership currently includes representatives of 14 member agencies. These member agency representatives will be joined by all statistical officials designated pursuant to section 2(b) of this guidance that are not already members, increasing ICSP membership by 12. Consequently, the ICSP will include two members each from the Department of Commerce and the USDA, which each contain two principal statistical agencies, and one member each from all other CFO Act agencies.

The expanded membership will augment the ICSP's ability to meet its statutory purpose. In addition, it will be a forum for collaboration, coordination, and information sharing among the statistical agencies and additional statistical programs across all member agencies, including on issues such as ensuring data quality and confidentiality, attaining and providing data access, and playing an effective role in agency-wide data governance.

Evaluation Officer Council The 0MB Evidence Team will establish and coordinate an interagency Council composed of Evaluation Officers. The Evaluation Officer Council will serve as a forum to exchange information, consult with and advise 0MB on issues that affect evaluation functions (e.g., evaluator competencies, program evaluation best practices, and evaluation capacity-building), coordinate and collaborate on areas of common interest, and serve in a leadership role for the broader Federal evaluation community. More information on the Council's composition and membership is forthcoming.

ID. Conduct Assessments and Planning

(a) Annual Evaluation Plan CFO Act agencies shall immediately begin the process of developing their first Annual Evaluation Plan. Non-CFO Act agencies, as well as sub-agencies, operational divisions, and bureaus of CFO Act Agencies are strongly encouraged to develop an evaluation plan as appropriate. At a minimum, the plan shall describe evaluation activities for the subsequent year, including the key questions for each planned "significant" evaluation study, as well as the key information collections or acquisitions the agency plans to begin.21 Below are the required activities by which agencies will document progress in developing and publishing an Annual Evaluation Plan:

20 44 U.S.C.

§ 3504(e). 21 The significance of an evaluation study should be defined by each agency and take into consideration factors such as the importance of a program or funding stream to the agency mission; the size of the program in terms of funding or people served; and the extent to which the study will fill an important knowledge gap regarding the program, population(s) served, or the issue(s) that the program was designed to address. Agencies should clearly state their criteria for designating evaluations as "significant" in their plan. See 5 U.S.C. § 312(b).

10

• FY 2021 Budget Submission (September 2019): Concurrent with their FY 2021 Budget submissions in September 2019, agencies must document their progress in creating an Annual Evaluation Plan in the Evidence Template that agencies are required to complete per 0MB Circular No. A-11.

• FY 2022 Annual Performance Plan Submission (September 2020): Concurrent with the submission of their FY 2022 Annual Performance Plan in September 2020 (to coincide with the submission of the FY 2022 Budget), agencies must submit an Evaluation Plan that covers evaluation activities planned for FY 2022. Following this submission, agencies are expected to provide an Annual Evaluation Plan concurrent with the submission of their Annual Performance Plan to 0MB in September of each year, which will be published along with the performance plans the following February.

(b) Capacity Assessment CFO Act agencies should immediately begin to plan how they will assess the coverage, quality, methods, effectiveness, and independence of their statistics, evaluation, research, and analysis efforts. Non-CFO Act agencies, as well as sub-agencies, operational divisions, and bureaus of CFO Act agencies are strongly encouraged to plan and conduct a capacity assessment as appropriate. This assessment should provide a comprehensive view of agency capacity, while also ensuring that the review takes into account specific needs identified through the development of the agency's learning agenda. Below are the key activities for agencies to document progress in developing and undertaking a capacity assessment:

• FY 2021 Budget Submission (September 2019): Concurrent with their FY 2021 Budget submissions in September 2019, agencies should propose an approach to conducting the assessment in the Evidence Template that agencies are required to complete per 0MB Circular No. A-11.

• FY 2022 Annual Performance Plan and Budget Submission (September 2020): Concurrent with their FY 2022 Annual Performance Plan submission in September 2020 (to coincide with the submission of the FY 2022 Budget), agencies must submit an interim capacity assessment documenting information gathered to date and outlining steps the agency intends to carry out to complete their full assessment.

• Draft Strategic Plan (May, September, and December 2021): As part of their initial draft Strategic Plan submitted to 0MB in May 2021, agencies must submit an initial draft capacity assessment that builds on and revises, as needed, the interim assessment submitted in September 2020. The full draft capacity assessment must be submitted in September 2021 with the full draft Strategic Plan. The final draft capacity assessment must be submitted in December 2021 as part of the agency's final draft Strategic Plan.

• Final Strategic Plan (February 2022): As part of their final Strategic Plan posted publicly in February 2022, agencies must include thefr updated, final capacity assessment. 11

(c) Open Data Plan The Evidence Act requires each agency to develop and maintain an Open Data Plan, 22 which, in general, shall describe the agency's efforts to make government data open to the public. The Open Data Plan shall be included in the agency's Strategic Information Resources Management Plan, which describes how information resource management activities help accomplish agency missions, including how they support government-wide financial management priorities, to include better integration of budget, cost, and performance data.

These plans shall be updated annually and made publicly available on the agency's website not later than five days after each update.23 OMB's forthcoming Phase 2 Guidance will provide agencies further guidance necessary to implement the Evidence Act's Open Data Plan requirement.

The Open Data Plan is one approach through which the Evidence Act strengthens the Federal government's commitment to open data. Many of the other open data requirements in the Evidence Act will be familiar to agencies because of existing open data responsibilities articulated in 0MB Circular No. A-130 (Managing 1,iformation as a Strategic Resource) and 0MB M-13-13 (Open Data Policy-Managing Information as an Asset). While the Evidence Act, in significant respect, carries forward the existing mechanisms for meeting open data policy objectives, the forthcoming Phase 2 Guidance will substantively update technical aspects of operationalizing these mechanisms. Until the Phase 2 Guidance goes into effect, agencies are reminded to continue meeting their existing open data obligations.

Appendices

Appendix A: Components of Evidence

Appendix B: Further Guidance on Learning Agendas

Appendix C: Further Guidance on Constituting Data Governance and Leadership

Appendix D: Further Guidance on Assessments and Planning

Appendix E: Implementation Resources for Agencies

22 44 U.S.C.

§ 3506(b)(2).

23 See 0MB Circular No.

A-130, § 5.a. 12

APPENDIX A: COMPONENTS OF EVIDENCE

Figure A.1 depicts and describes four interdependent components of evidence: foundational fact finding, policy analysis, program evaluation, and performance measurement. Each of these components informs and directs the others, and many evidence-building activities may be hard to categorize because they organically include more than one component. As agencies build and use evidence, per the Evidence Act and this guidance, they should consider these various types of evidence.

Figure A. 1: Components of Evidence Policy Analysis Analysis of data, such as general purpose survey or program-specific data, to generate and inform policy, e.g., estimating regulatory impacts and other relevant effects Foundational Fact Finding Foundational research and analysis such as aggregate indicators, exploratory studies, descriptive statistics, and basic research Program Evaluation System atic analysis of a program , policy, organization, or component of these to assess effectiveness and efficiency Performance Measurement Ongoing, systematic tracking of information relevant to policies, strategies, programs, projects, goals/objectives, and/or activities 13

APPENDIX B: FURTHER GUIDANCE ON LEARNING AGENDAS

I. Purpose The Evidence Act requires that agencies' strategic plans include a section on evidence building to be developed in conjunction with the agency's process of updating its strategic plan every four years. These evidence-building plans are systematic plans for identifying and addressing priority questions relevant to the programs, policies, and regulations of the agency. Thus, evidence­ building plans are multi-year learning agendas that provide an evidence-building roadmap to support effective and efficient agency functioning. Learning agendas offer the opportunity to use data in service of addressing the key questions an agency wants to answer to improve its operational and programmatic outcomes and develop appropriate policies and regulations to support successful mission accomplishment. They identify, prioritize, and establish strategies to develop evidence to answer important short-and long-term strategic questions (i.e., questions about how the agency meets its mission(s), including about how programs, policies, and regulations function both individually and in combination) and operational questions (i.e., questions about the agency's operations like human resources, grant-making procedures, financial systems and tracking, and internal processes).24 In this context, such strategic and operational questions to be answered can subsequently be integrated into and reflected in the agency's Strategic Plan and Annual Performance Plan. When items in the learning agenda are designed to support planned regulatory actions, the agency should cross reference the entry for that regulatory action in the current Unified Agenda of Federal Regulatory and Deregulatory Actions.

Learning agendas should be iterative, flexible, transparent, and tailored to both meet an individual agency's needs and address agency-specific challenges to developing evidence. They should highlight recent, existing, and future evidence-building activities; make an agency's evidence-building plans transparent; and promote interest in and support for the studies, evaluations, and other evidence that will follow from them. Through the necessary consultations with relevant stakeholders, learning agendas provide an opportunity to align efforts and promote interagency collaboration in areas of joint focus or shared populations or goals. For agencies with emerging evidence-building functions, the learning agenda may emphasize and help coordinate efforts to design and implement new evidence-building activities. For agencies with more mature evidence-building efforts that may already have comprehensive research plans, research roadmaps, enterprise learning agendas, or evaluation strategic plans in place, the learning agenda may highlight and prioritize existing efforts. As a result, there is no single approach or format for a learning agenda that will be effective for every agency.

A learning agenda is a systematic way to identify the data agencies intend to collect, use, or acquire, as well as the methods and analytical approaches to facilitate the use of evidence in policymaking. It promotes deliberate planning of evidence-building activities, including how to prioritize limited resources and address potential information gaps that may inhibit an agency's effective mitigation of risks identified through their enterprise risk management processes.

Leaming agendas relate to, but are distinct from research and development planning for the 24 While a learning agenda may address an agency's human resources as those resources relate to an identified priority question, learning agendas do not refer to plans for human capital development.

purposes of building the stock ofknowledge.25 They can also complement an agency's submission to the Unified Agenda of Federal Regulatory and Deregulatory Actions by clearly documenting how an agency intends to build and use evidence to support its proposed regulatory actions.

II. Establishing and Utilizing a Multi-Year Learning Agenda Requirements for a Learning Agenda The Evidence Act requires CFO Act agencies26 to have a learning agenda that covers a four-year period aligned with the strategic plan and that addresses priority questions (i.e., questions relevant for programmatic, operational, regulatory, or policy decision-making) across the entire agency (i.e., the entire Cabinet-level Department). Additionally, 0MB strongly encourages non­ CFO Act agencies, as well as sub-agencies, operational divisions, or bureaus of CFO Act agencies to develop and implement their own learning agendas that tie into and are consistent with, but are not limited by, the agency-wide agenda.

Management of the Development and Implementation Process Pursuant to the Evidence Act, the agency's designated Evaluation Officer (see Appendix C) shall coordinate, develop, and implement the learning agenda. The Evaluation Officer shall play a leading technical role in the development and implementation of the learning agenda at the agency level and support efforts to develop learning agendas at the sub-agency, operational division, or bureau levels, particularly as those agendas relate to the agency-wide plan. Other senior officials who work on evidence-building activities, such as the Statistical Official and Chief Data Officer, may also play an important role in the learning agenda process. Agency leadership should be engaged in the development and implementation of the learning agenda, as their buy-in and support for determining what high-priority questions should be included, which stakeholders should inform the process, and which staff and data assets will be needed to fulfill the learning agenda, are critical to success.

Key Components of the Learning Agenda Process 0MB recognizes that there is not just one way to develop a learning agenda. Agencies should undertake the learning agenda process in a way that is most appropriate to their particular agency. To meet the requirements for learning agendas outlined in Section 312 of the Evidence Act, agencies must include the components described further below in the process. Figure B.1

(below) shows one potential way that agencies could order these components to develop a learning agenda, though there are other approaches, and agencies may find that a different process best meets their needs.

25 Pursuant to 0MB Circular No.

A-11, Section 210.13, Research and Development programs "focus on knowledge creation or its application to the creation of systems, methods, materials, or technologies, such as DO E's Solar Energy and NASA's Solar System Exploration programs."

26 5 U.S.C.

§ 312(a). 15

Figure B.1: Example Learning Agenda Process a ; Engage Update Stakeholders Learning on Agency Agenda Learning I Priorities \ Use Results identify to Inform Agency Priority Efforts Questions I Undertake Write the Learning Learning Agenda p • Agenda Activities Engage Stakeholders on Agency Learning Priorities. Agencies should consult with internal and external stakeholders throughout the process to develop and implement a learning agenda as learning priorities are identified and evidence-building activities are conducted. The purpose of stakeholder engagement is to make sure that the learning agenda addresses questions that are relevant, salient, and meaningful to those with direct interests in the agency's functions, and that / the learning that results resonates with stakeholders. Agencies should consult with internal agency stakeholders who play a role in, have a stake in, or will use the results of the learning process, including agency staff who oversee the designs, processes, operations, or programs being discussed in the plan; other evaluation, statistics, analysis, data, enterprise risk management , and performance units and personnel in the agency; policy staff; regulatory staff; privacy and information law and policy personnel; and agency leadership. Per the Evidence Act, agencies shall also consult with the public, State and local governments, and representatives of non-governmental researchers. 0MB also urges broader consultation with additional stakeholders, including: the Office of Management and Budget, Federal grant recipients, Congress, and industry and trade groups. Insofar as an agency's learning agenda describes its plans to build evidence to address specific questions or fill certain purposes, such as supporting data-driven, performance and strategic review processes and routines, these agendas also signal agency priorities externally, including opportunities for collaboration with external stakeholders to answer some priority questions. The agency's learning agenda should describe its stakeholder engagement, including the process and participants.

Agencies should gather input in the manner that best meets their needs, most effectively engages their specific stakeholders, and leverages existing activities and/or requirements whenever possible, in accordance with applicable law and policy. Potential models for doing so include: Requests for Information published in the Federal Register, listening sessions with groups of 16

stakeholders, Technical Working Groups, and one-on-one consultations. 0MB recognizes that agencies may use different approaches at different points in the process, and that it may not be feasible to engage all stakeholders for all updates to the learning agenda. While agencies have the option of soliciting feedback and engaging specific stakeholders on their learning agenda as part of their consultation process on the Agency Strategic Plan27 , agencies may choose to conduct separate consultations to account for varied interests of different stakeholder groups as it relates to these components of the agency's Strategic Plan.

Identify Priority Questions. Agencies should identify priority questions that, when answered, will have the biggest impact on agency functioning and performance. Agencies may, but are not required to, tie their questions to strategic goals and strategic planning. Learning agendas should include both short-and long-term questions of interest to the agency, as well as mission-strategic and agency-operational questions (see Table B.1 below), in a balance that emphasizes those of greatest relevance and priority to the agency. In identifying questions, agencies should move beyond high-level, broad questions, even at the enterprise level, to those that have sufficient detail to be answerable and useful.

Mission - Strategic Agency­ Operational

• I • : I ' I I I ' Short-Term negatively predict increased numbers of participants that complete Agency A's Program B's sessions? - How can we prospectively monitor and evaluate county-level interventions and initiatives funded by Program C in Agency D? - What factors are critical for meeting Agency J's sub-contracting goals?

- What effect did revised procedures have on Agency K's ability to hire new staff within 120 days? What was the impact of online eligibility forms vs. paper forms for time to program enrollment for target population L? - How do Agency M's policies encourage grant recipients to implement practices to support stronger grant performance?

27 See 0MB Circular No.

A-11 , § 230.15. 17 Long-Term - Have regulatory actions reduced outcome E in program area F for programs funded by Agency C? - Have capacity-building and increased opportunities for target population G led to increased participation of target population Gin intervention H?

- Based on projected demographic changes and economic forecasts, how will demand for Program I likely change, and what new challenges will these changes pose? - What regulatory, policy, or process improvements could be made to help strengthen the agency's oversight and risk management?

- How can Bureau N in Agency P improve partnerships with local organizations? How can Agency Q ensure that its workforce has the ongoing IT skills needed to meet Agency R's mission?

In defining these questions, agencies should articulate how they define short-and long-term (e.g., short-term is less than one year, while long-term is over the next four years), and the period over which they expect to answer these questions. Learning agendas can give agencies the opportunity to strategically build evidence via systematic questions and agencies should, to the extent possible, describe how short-and long-term questions relate to one another. For instance, an agency may want to understand the impact on participants of a large-scale grant program. In order to answer this long-term question, the agency may need to answer a series of short-term questions, like "How do grantees implement the program in individual locations?" or "To what extent do program participants achieve the desired intermediate-term program outcomes?"

Agencies should consider and include the range of methods and types of evidence that could be brought to bear to answer the priority questions identified in their learning agenda, including foundational fact finding, performance measurement, policy analysis, and program evaluation (see Appendix A). Agencies, to the extent practicable, may also answer questions using existing evidence, including literature reviews, meta-analyses, and research clearinghouses. Indeed, existing evidence is critical to identify gaps in knowledge and unmet needs, and learning agendas should address the current state of evidence. The questions to be answered must drive the research methods (and not vice versa), and agencies are expected to use the most rigorous' methods possible that align to the identified questions and are feasible in a given circumstance.

Agencies should, thus, be realistic about what can be measured and what cannot when prioritizing questions. As appropriate, agencies are encouraged to discuss opportunities for external entities to support the agency's learning agenda, including by explicitly identifying questions that are amenable to such collaboration.

Write the Learning Agenda. Written learning agendas should include the feedback gathered from the agency's stakeholder engagement efforts and use clear, plain language. Consistent with the Strategic Plan, agencies should make their learning agenda public (e.g., by posting on their agency website and disseminating to stakeholders), though agencies should determine what is most appropriate to publish based on their own needs, content, and relevant statutory requirements. At a minimum, agencies should include the following types of content in their learning agenda, recognizing that this content does not necessarily need to be organized according to these headings:

• Which Strategic Goals and Objectives that the Learning Agenda will Address;

• Priority Questions to be Answered;

• Activities that the Agency will Engage in to Address Priority Questions;

• Timing of Learning Agenda Activities;

• Potential Data, Tools, Methods and Analytic Approaches to be Used to Answer Priority Questions; and

• Anticipated Agency-Specific Challenges and Proposed Solutions to Developing Evidence to Support Agency Priorities. 18

Undertake Learning Agenda Activities. Agencies should carry out the activities specified in the learning agenda, including conducting evaluations (internally or extemally),28 data analyses, performance activities, syntheses of existing evidence, etc. In doing so, agencies are expected to abide by relevant 0MB guidance.29 It will be important for agencies to consider ways to combine or coordinate related evidence and data-related activities, in accordance with applicable law governing the collection, use, and disclosure of data and information.

Use Results to Inform Agency Efforts. Critical to the learning agenda process is the actual learning that results from the evidence-building activities an agency undertakes and how the agency uses and disseminates that information. Agencies should disseminate results internally to the appropriate parties for learning, improvement, and decision-making. As agencies consult with internal and external stakeholders, they should consider how to best time dissemination to maximize the usefulness of the evidence. In addition, agencies should have a plan for external publication of findings from program evaluations, research studies, and related analyses per accepted standards of scientific integrity as noted in the agency's evaluation policy or other procedures. Beyond dissemination, agencies should also consider how they will use results to inform agency decisions and activities, including, for example, continuous quality improvement, strategic planning, and budget formulation. For example, agencies can leverage their internal strategic review process, a key management routine for ensuring agency implementation strategies and activities can be updated, to improve strategic goals, objectives, and programmatic outcomes. Agencies must ensure that the use of the results of leaI'J).ing activities is done in accordance with applicable law and policies governing the collection, use, and disclosure of data and information.

Update Learning Agenda . Updating the learning agenda builds and maintains the culture of learning an agency needs in order to think critically and to examine and systematically improve programs, processes, regulations, and policies. Agencies must revisit their learning agendas at least annually and update them as needed to reflect progress toward meeting the agency's original goals and objectives, shifting agency priorities, changing contexts within which the agency operates, and emergent needs. Leaming agendas should also be updated to incorporate, when available, the results of the activities an agency undertakes to answer priority questions.

0MB does not expect that agencies will rewrite or draft a new learning agenda annually. Rather, the focus should be on creating meaningful updates as appropriate. 28 Significant evaluations undertaken as part of learning agenda implementation should be included in the annual E valuation Plans required by the Evidence Act.

29 See Guidelines for E nsuring and Maximizing the Quality, Objectivity, Utility, and Integrity oflnformation Disseminated by Federal Agencies; Office ofMgm 't & Budget, Exec. Office of the President, 0MB M-05-03 , Final Information Quality Bulletin for Peer Review (2005), available at https://www .whitehouse.gov/sites/whitehouse. gov/files/omb /assets/OMB /inforeg/igg oct2002.pdf; 0MB M-14-06 ; principles and best practices in the Federal Data Strategy (forthcoming); and future 0MB guidance on program evaluation.

19

APPENDIX C: FURTHER GUIDANCE ON CONSITUTING DATA GOVERNANCE AND LEADERSHIP

Implementing a Data Governance Body I. Purpose Data governance is the process of setting and enforcing priorities for managing data as a strategic asset in service of an agency's mission, including its strategic plan and other high-level priorities. This is accomplished by establishing a data governance structure with sufficient authority over the management of and access to key data assets, including roles for developing, overseeing, and coordinating data management policy and resource allocation.

Data governance is foundational to undertaking many activities required by the Evidence Act, and responsibility is shared among multiple parties. Effective implementation of data governance therefore requires a data governance body that, at a minimum:

• Supports agency priorities, such as those identified in strategic plans and learning agendas, by informing Strategic Information Resources Management Plans for the agency and identifying resources to implement those priorities;

• Coordinates and supports implementation of data management responsibilities with agency data-management actors to strengthen strategic plans and learning agendas;

• Sets agency data policy in a manner that complements, but does not supplant the statutory authority of established positions; and

• Coordinates implementation of the Federal Data Strategy by assessing data maturity, risks, and capabilities to recommend related data investment priorities. By September 30, 2019, the head of each agency must establish an agency Data Governance Body, chaired by the Chief Data Officer, with participation from relevant senior-level staff in agency business units and data support functions and regular meetings.

II. People An enterprise-wide Data Governance Body is only effective when the agency head or delegated authority sponsors the program and assigns senior staff with responsibility for ensuring that the data governance program addresses key agency concerns and priorities. In addition to the Chair (i.e., the Chief Data Officer), an agency's Data Governance Body should include appropriate senior-level staff and technical experts needed to discuss and set policy on a range of data and data-related topics.

Agencies are encouraged to constitute their Data Governance Bodies in a way that ensures that they meet the requirements discussed below for that particular agency. At a minimum, agencies should include the newly-designated positions from the Evidence Act (i.e., the Chief Data Officer, Evaluation Officer, and Statistical Official) and senior officials from agency program areas. In addition, 0MB expects that the following positions have defined roles in the 20

governance process and may be members of the Body: Chief Acquisition Officer, Chief Financial Officer, Chief Freedom oflnformation Act Officer, Chieflnformation Officer, Chief Information Security Officer, General Counsel, Performance Improvement Officer, Senior Agency Official for Geospatial Information, Senior Agency Official for Privacy, and Senior Agency Official for Records Management. Agencies may also augment their Data Governance Body with other relevant senior officials, such as Chief Operating Officer and Chief Technology Officer for permanent or rotational participation in accordance with agency governance rules and customs. The makeup of an agency's Data Governance Body should be driven by the agency's needs and structure.

III. Reporting All agencies must include their Data Governance Body in their Strategic Information Resources Management Plans' Governance sections as required by 0MB guidance.30 Additionally all agencies must report their Data Governance Body members to 0MB via email at EvidenceAct@omb.eop.gov no later than September 30, 2019.

Key Senior Officials The Evidence Act requires agencies to designate three senior officials - Chief Data Officer, Evaluation Officer, and Statistical Official - who will oversee the use of data and evidence­ building activities in agencies. The following outlines OMB's expectations for the designation of these officials, as well as expectations for how these individuals will work with one another within agencies. 31 Chief Data Officer I. Background While there are many roles in the Federal Government that relate to data management, over the last few years, Chief Data Officers (CDOs) have emerged to lead organizational development of processes to leverage the power of data. CDOs enable data driven decision-making in a variety of ways, from providing and leveraging centralized agency analytics capacity32 to creating tools and platforms that enable self-service across their agencies and for the public.33 Successful data management must account for every stage of the data lifecycle. Among other things, it involves establishing effective procedures, standards, and controls to ensure quality, accuracy, access, and protection of data, as well as managing information technology and information security. The

30 See 0MB Circular No.

A-130 , § 5.a. 31 As illustrated in the descriptions of functions and responsibilities for each of these positions, it is OMB 's expectation that these are full-time positions. Agencies should identify and designate staff accordingly.

32 See Office of Mgm 't & Budget, Exec.

Office of the President, 0MB M-19-16 , Centralized Mission Support Capabilities for the Federal Government (2019), available athttps://www.whitehouse.gov/wp-content/uploads/20 l 9/04/M-19-16.pdf.

33 Jane M .

Wiseman , Ash Ctr. for Democratic Governance & Innovation, Harvard Kennedy Sch. Data-Driven Government: The Role of Chief Data Officers. (2018), available at https://www .innovations.harvard.edu/sites/default/files/Data-Driven­ Government. pdf. 21

Evidence Act formally establishes within agencies the position of the CDO, who, through the functions described below, and in coordination with other agency leaders, will ensure the successful execution of the agency's data management responsibilities. II. Designation of Chief Data Officers By July 13, 2019, the head of each agency shall designate a nonpolitical appointee employee in the agency as a Chief Data Officer to support implementation of the Evidence Act and serve on the Chief Data Officer Council. Agency heads shall designate this role based on the qualifications below and place this role in the best position within their agency to serve the functions listed below.

The Evidence Act does not require this position to be within any particular organizational unit of the agency or to report to any particular official. However, consistent with the role they play in the agency, CDOs shall serve in a central leadership position, with visibility into relevant agency operations, and be positioned highly enough to regularly engage with other agency leadership, including the head of the agency. Agencies, as part of their Information Resource Management governance planning activities, shall evaluate the effectiveness of the organizational placement of roles and the allocation of responsibilities.

The CDO shall have the necessary authority to carry out the responsibilities described below. Nevertheless, some of these responsibilities are shared with others in the agency. Therefore, these functions should be carried out in consultation with the Data Governance Body and agency officials who also may have shared responsibilities under other statutes. The agency head is responsible for assuring that shared responsibilities are carried out in an effective and efficient manner.

III. Qualifications for Chief Data Officers CDOs must have the necessary skills, knowledge, and expertise to carry out the functions assigned by law and described below. They should be experts in data management and governance (including creation, application, and maintenance of data standards) and have a working knowledge of data collection, analysis, protection, use, and dissemination as it relates to the agency's mission.34 Specific qualifications include:

• Demonstrated ability to communicate effectively, build coalitions, and exert influence with political savvy among stakeholders within large organizations, between organizations and government entities (e.g., between state and Federal government bodies), and among data users and providers;

• Demonstrated ability to successfully manage organizational culture change; 34 According to section 202(e) of the Evidence Act, "The Chief Data Officer of an agency shall be designated on the basis of demonstrated training and experience in data management, governance (including creation, application, and maintenance of data standards), collection, analysis, protection, use, and dissemination, including with respect to any statistical and related techniques to protect and de-identify confidential data." 44 U.S.C. § 3520.

22

• Demonstrated ability to facilitate and implement best practices in data curation and management across the entirety of the data lifecycle;

• Demonstrated ability to facilitate improved regulatory outcomes by supporting agency compliance with Executive Order 12866, Regulatory Planning and Review, and 0MB Circular No. A-4, Regulatory Analysis;

• Demonstrated ability to think strategically and solve complex problems regarding the application of data to data-driven decision-making;

• Demonstrated ability to facilitate complex data analysis to improve operational results, regulatory activities, and mission outcomes; and

• Strong business acumen and familiarity with data science approaches, cloud computing, cyber security, privacy, confidentiality, data analytics, statistical methods, policy analysis, information quality standards set out in the Information Quality Act (IQA),35 record disclosure laws and policy, and the Paperwork Reduction Act (PRA).

IV. Responsibilities for Chief Data Officers Governance Responsibilities One of the CDO's most important responsibilities is to convene and coordinate agency-wide data governance, which is a shared activity among multiple data stewards and supporting officials. In this regard, the Evidence Act assigns specific functions to the CDO, but it is OMB's expectation that other agency leaders continue to perform the functions assigned to them by relevant statutes.

The head of the agency retains ultimate responsibility for the success of the agency's data management activities. To further the necessary collaborative environment, the CDO's convening and coordination functions shall include:36

• Leading the agency's Data Governance Body, including facilitating collaborative activities among the numerous actors with responsibilities and needs for data within the agency;

• Supporting the agency's learning agenda by coordinating data access and management activities that support evidence building, including evaluation, performance reporting, and the analysis of regulations;

• Coordinating with any official in the agency responsible for using, protecting, disseminating, and generating data to ensure that the data needs of the agency are met;

• Engaging agency employees, the public, and contractors in using public data assets, and encouraging collaborative approaches on improving data use;

• Supporting the Performance Improvement Officer of the agency in identifying and using data to carry out the functions described in 31 U.S.C. § 1124(a)(2);

• Supporting the Evaluation Officer of the agency in obtaining data to carry out the functions described in 5 U.S.C. § 313(d);

35 Id.

§ 3516, note. 36 Section 202(e) of the Evidence Act delineates functions of the Chief Data Officer. See id. § 3520(c). 23

• Reviewing the impact of the infrastructure of the agency on data asset accessibility and coordinating with the Chief Information Officer of the agency to improve such infrastructure to reduce barriers that inhibit data asset accessibility;

• Ensuring that, to the extent practicable, the agency maximizes its use of data, including for the production of evidence, including regulatory analyses, cybersecurity, and the improvement of agency operations;

• Identifying points of contact for roles and responsibilities related to open data use and implementation;

• Serving as the agency liaison to other agencies and 0MB on the best way to use existing ' agency data for statistical purposes. This function includes helping program units provide data to statistical agencies and units, as required in section 303 of the Evidence Act;37 and

• Complying with any regulation and guidance issued under the newly added subchapter III of chapter 35 of title 44 of the United States Code, including the acquisition and maintenance of any required certification and training. Lifecycle Data Management Responsibilities The second major area of responsibility for a Chief Data Officer is the agency's lifecycle data management, including:38

• Ensuring that the agency follows, to the extent practicable, data management best practices;39

• Carrying out information resources management strategic planning,40 including leading the development of the agency's Open Data Plan;41

• Carrying out the requirements of the agency with regard to the information collection and control of paperwork, including certifying to 0MB the agency's compliance with 5 CFR 1320.9 and 1320.8(b)(3);42

• Carrying out information dissemination;43

• Managing the agency's data assets, including the standardization of data format, sharing of data assets, and publication of data assets in accordance with applicable law;44 and

• Carrying out records management. 45 Delegation of Responsibilities for Statistical Law The protection of information collected for statistical purposes is a core duty of the statistical agencies; the ability of statistical agencies to fulfill this duty is contingent on compliance with all laws and regulations that they are bound to follow. The CDO must delegate the above responsibilities to the head of a principal statistical agency or recognized statistical unit to ensure

37 Id.

§ 3581.

38 Id.

§ 3520(c)(l).

39 Id.

§ 3520(c)(6).

40 Id.

§§ 3506(b), 3520(c)(5).

41 Id.

§§ 3506(b)(2), 3511.

42 Id.

§§ 3506(c); 3507.

43 Id.

§ 3506(d).

44 Id.

§ 3520((c)(3).

45 Id.

§ 3506(f). 24

compliance with laws and associated policies specific to statistical agencies, including section 314 of title I and all of title III of the Evidence Act, as well as policies promulgated pursuant to

44 U.S.C.

§§ 3504(e) and 3506(e) and the statutes authorizing each of the principal statistical agencies and each of their statistical programs. Statistical agency or unit heads, to the extent permissible by law, must consult with the CDOs in carrying out said responsibilities, and CDOs must defer to them with respect to any data acquired, maintained, or disseminated by the agency under applicable statistical law.46 Evaluation Officer I. Background Strengthening the program evaluation function within the Federal government is an important step in expanding evidence building. The Evaluation Officer shall be responsible for overseeing the agency's evaluation activities, learning agenda, and capacity assessment, as well as collaborating with, shaping, and contributing to other evidence-building functions within the agency. The Evaluation Officer is responsible for providing technical and methodological leadership to assess, improve, and advise on evaluation activities across the agency. For agencies that are less mature in their evaluation activities, or for those agencies without additional evaluation expertise distributed throughout the agency, the Evaluation Officer may also be responsible for conceptualizing, prioritizing, and designing the agency's evaluation activities.

II. Designation of Evaluation Officers By July 13, 2019, the head of each of the 24 Federal agencies covered by the CFO Act must designate a senior employee of the agency as the Evaluation Officer based on demonstrated technical expertise and skills in program evaluation methodology and practices and without regard to political affiliation.47 Each agency must have one Evaluation Officer to serve in this role for the agency as a whole (e.g., the entire Cabinet-level department). In addition, where appropriate, 0MB strongly recommends that agencies consider building capacity in sub­ agencies, operational divisions, or bureaus within the agency that also may have a need for an Evaluation Officer, with the agency-level official providing expert leadership and coordination among those officials.

Agencies have the flexibility to designate their Evaluation Officer based on the agency's organizational structure and particular needs. When designating the Evaluation Officer, agency heads should consider lines of accountability and the coordination needed to ensure the Evaluation Officer is aligned and partnered with supporting offices and functions to effectively perform their roles and responsibilities. Regardless of structure, there must be one Evaluation Officer at the agency-wide level who facilitates and oversees evaluation activities across the agency. It is important to note that the Evaluation Officer cannot be a member of the Inspector

46 Id.

§ 3520(d)(3). 47 All other agencies are encouraged to designate Evaluation Officers and implement this guidance. 25

General's staff or located in the Inspector General's (IG) office. While IGs play a very important role in agency oversight and accountability, the Evaluation Officer required by the Evidence Act has a different function. III. Qualifications for Evaluation Officers The Evaluation Officer shall be appointed without regard to political affiliation based on demonstrated, senior-level technical expertise in evaluation methods and practices and documented expertise to the disciplines of the agency. This includes advanced education and training in an evaluation-related field, direct experience in applying evaluation methods to study Federal programs and/or policies, and knowledge of current analytic techniques. Agencies shall ensure that they appoint or designate an Evaluation Officer with credibility in the range of relevant evaluation practices and methods, as well as appropriate expertise in the culture, disciplines, and policy areas of the agency. The Evaluation Officer must have the standing and authority to perform the functions listed below. Continuity of their leadership and the ability to oversee multi-year efforts are likewise important. Evaluation Officers should also have demonstrated ability to:

• Ensure that the agency maintains principles of scientific integrity throughout the evaluation process, including adherence to professional values and practices when conducting and using the findings from evaluation to maintain objectivity, clarity, and reproducibility and to insulate from bias, fabrication, falsification, censorship, or other threats to integrity;

• Ensure that rigorous evaluation is a priority that is not subordinated to other agency demands;

• Communicate effectively, build coalitions, and exert influence with political savvy among stakeholders within large organizations, between organizations and government entities (e.g., between state and Federal government bodies), and among data users and providers;

• Think strategically about allocation of evaluation resources and dissemination of evaluation findings to support agency business and mission needs;

• Communicate the findings of evidence-building activities to a range of stakeholder audiences; and

• Successfully manage organizational culture change. The Evaluation Officer must report to a key agency official so that evaluation activities, including findings, can be integrated into decision-making processes. The Evaluation Officer must devote sufficient time to this role to meet the expectations and functions outlined in section 313 of the Evidence Act and reflected in this Memorandum.

26

IV. Responsibilities for Evaluation Officers The Evaluation Officer shall have authority and responsibility for providing leadership over the agency's (or sub-agency's, operational division's, or bureau's) evaluation activities, and will be expected to perform the following functions:

• Serve as: o Agency champion for and educator of agency staff and leaders about evaluation, including what evaluation is, the value of conducting evaluations, how to discern high-quality evaluation from other types of analyses, and the importance of evaluation as a strategic investment; o Senior advisor to agency leaders on issues of evaluation policy and practice, such as designing and undertaking evaluations, interpreting results, and integrating evaluation findings into day-to-day agency operations, management processes, budgeting, strategic planning, and other decision-making; o Senior agency contact on evaluation for agency-wide and cross-cutting evaluation efforts, both with external stakeholders and in coordination with senior officials responsible for other agency functions, including officials responsible for implementing privacy policy, the Chief Data Officer, the Chieflnformation Officer, the Statistical Official, the Performance Improvement Officer, additional evaluation and analysis units and personnel in the agency, and others as appropriate; o Participant in the agency's Chief Operating Officer-led efforts to review progress on Agency Priority Goals and other management priorities to ensure that evidence is included and used appropriately; o Member of the agency Data Governance Body; and o Member of the interagency Evaluation Officer Council.

• Oversee or conduct: o Assessment of the coverage, quality, methods, effectiveness, objectivity, scientific integrity, and balance of the portfolio of evaluations, policy research, and ongoing evaluation activities of the agency, in consultation with other methodologists, such as the Statistical Official, when appropriate; o Improvement of agency capacity to support the development and use of evaluation, coordinate and increase technical expertise available for evaluation and related research activities within the agency, and improve the quality of evaluations and knowledge of evaluation methodology and standards; o Implementation of the forthcoming 0MB guidance for program evaluation standards and best practices; o Establishment and implementation of an agency evaluation policy that affirms the agency's commitment to conducting rigorous, relevant evaluations and to using evidence from evaluations to inform policy and practice. The policy will provide the agency's stakeholders with a clear understanding of the expectations related to key principles, such as evaluation rigor, relevance, transparency, independence, and ethics; o Coordination, development, and implementation of the plans required under section 312 of the Evidence Act: 27

• Learning Agenda ( evidence-building plan) in conjunction with the Agency Strategic Plan;  Annual Evaluation Plan, submitted in conjunction with the Annual Performance Plan; and

• Capacity Assessment, included as part of the Agency Strategic Plan. o Development of new or improvement of existing processes to integrate evaluation findings into agency decision-making and other functions; o Management of agency-wide evaluation standards and requirements to ensure the scientific integrity of the agency's evaluation activities; and o Use and dissemination of evaluation results throughout the agency and to the public, as appropriate.

Statistical Official I. _ Background Statistical expertise allows organizations to ensure that data are gathered, processed, and curated so as to produce statistical products with the highest standards of data quality while protecting confidentiality, privacy, and security. Data quality has multiple dimensions, including credibility and accuracy, timeliness and relevance to valuable decision-making processes, the objectivity with which data are produced, and their accessibility to multiple users at an appropriate level of clarity and detail. Applying statistical expertise involves the maintenance and development of policies that anticipate and address the needs of data users and providers, the continual advancement and adoption of statistical techniques to maximize the quality of statistical outputs, and the delineation of procedures to ensure that these policies and techniques are implemented in a rigorous and efficient manner.

II. Designation of Statistical Officials By July 13, 2019, the head of each of the 24 Federal agencies covered by the CFO Act must designate a statistical official to advise on statistical policy, techniques, and procedures.48 Within Federal agencies that contain one or more principal statistical agencies, the mantle of leadership to cultivate and apply statistical expertise is worn by those principal statistical agencies.

Statistical agencies have developed and demonstrated a high level of statistical expertise over many decades as the development and implementation of statistical surveys advanced and was expanded to incorporate a wide array of non-survey data from administrative, commercial, sensor-based, and other sources. They continue to lead as the field grows and develops methodologies to improve the quality and efficiency of statistical data uses. Statistical agency heads provide oversight and advice to the leadership of their parent agencies to promote and optimally balance the ideals of data quality and confidentiality. As champions of these ideals,

48 5 U.S.C.

§ 315(a) ("The head of each agency shall designate the head of any statistical agency or unit within the agency, or in the case of an agency that does not have a statistical agency or unit, any senior agency official with appropriate expertise, as a statistical official to advise on statistical policy, techniques, and procedures. Agency officials engaged in statistical activities may consult with any such statistical official as necessary.").

28

principal statistical agencies are a logical choice to play a broad role in overseeing the curation and use of their parent agencies' data assets. In agencies that do not contain principal statistical agencies, a variety of other advisory structures or personnel may play similar roles. The Evidence Act institutionalizes these roles by requiring each CFO Act agency to designate a Statistical Official. For agencies that house one of the 13 statistical agencies, the agency head must designate the head of a statistical agency as the Statistical Official. 49 Where one does not exist, an agency head should identify as a Statistical Official an individual who demonstrates high levels of expertise as described in the Qualifications section below. 0MB expects agencies to designate Statistical Officials based on the qualifications they possess and functions they are to perform as described in the Responsibilities section below, and that the individuals' qualifications shall take preference over other considerations. The ability of the Statistical Official to have continued leadership and the ability to oversee multi-year efforts are likewise important for agencies to consider as they designate this Official.

The Statistical Official must have the standing and authority to perform the functions listed in the Responsibilities section below. The Official must have sufficient independence to meet the requirements of the Evidence Act and to carry out the responsibilities of section 302(a) of the Evidence Act. so The Statistical Official must devote sufficient time to this role to meet the expectations and functions outlined in sections 101 and 302 of the Evidence Act51 and reflected in this Memorandum.

III. Qualifications for Statistical Officials Each Statistical Official must have appropriate expertise. This includes:

• Demonstrated ability to plan, implement, and direct large-scale ongoing and new statistical programs;

• Demonstrated experience in research and statistics as evidenced by published papers, presentations at professional meetings, and national recognition as an authority in a statistical field;

• Demonstrated ability to build and maintain productive relationships with high-level government policy officials, academics, scholars in research organizations, officials in international organizations, and stakeholders;

• Outstanding communication and analytical capabilities;

• Knowledge of Federal privacy and confidentiality laws and regulations as they relate to the curation of data records, the conduct of data collection activities, and the production of 49See id. The 13 principal statistical agencies are the Bureau of Economic Analysis (BEA) in the U.S. Department of Commerce; Bureau of Justice Statistics (BJS) in the U.S. Department of Justice; Bureau of Labor Statistics (BLS) in the U.S. Department of Labor; Bureau of Transportation Statistics (BTS) in the U.S. Department of Transportation; Census Bureau in the U.S.

Department of Commerce in the U.S. Department of Commerce; Economic Research Service (ERS) in the U.S. Department of Agriculture; Energy Information Administration (EIA) in the U.S. Department of Energy; National Agricultural Statistics Service

(NASS) in the U.S. Department of Agriculture; National Center for Education Statistics (NCES) in the U.S. Department of Education; National Center for Health Statistics (NCHS) in the U.S. Department of Health and Human Services; National Center for Science and Engineering Statistics (NCSES) in the National Science Foundation; Office of Research, Evaluation, and Statistics (ORES) in the Social Security Administration; and Statistics oflncome Division (SOI) in the Internal Revenue Service.

50 44 U.S.C.

§ 3563(a).

51 5 U.S.C.

§ 311 et seq.; 44 U.S.C. § 3561 et seq. 29

one-time and recurring statistical products;

• Broad and comprehensive technical knowledge of statistical principles, theory and methodology, and their application to issues in the agency's subject area; and

• Extensive knowledge of statistical programs and related functions and responsibilities of entities within the agency, as well as a thorough knowledge of statistical programs of other government agencies. IV. Responsibilities of Statistical Officials The Statistical Official will direct and coordinate statistical policy for the agency and will be expected to perform the following functions:

• Serve as the: o Agency champion for data quality to ensure data relevance ( e.g., by validating that data are appropriate, accurate, objective, accessible, useful, understandable, and timely), harness existing data (e.g., by identifying data needs and reusing data if possible), anticipate future uses (e.g., by building interoperability of data from its inception), and demonstrate responsiveness ( e.g., by improving data collection, analysis, and dissemination with ongoing input from users and stakeholders);52 o Agency champion for confidentiality protection and appropriate data access for agency data collected with legal or other restrictions on use or release;53 o Agency senior advisor for statistical policy, including establishing policies to enable agencies to produce and disseminate relevant and timely statistical information, conduct credible and accurate statistical activities, conduct objective statistical activities, and protect the trust of information providers by ensuring the confidentiality and exclusive statistical use of their responses;54 o Agency senior advisor for statistical techniques to ensure the use of techniques that provide optimal levels of data quality and confidentiality; and o Agency senior advisor for statistical procedures that implement desired policies and techniques systematically and efficiently. 55

52 See Office of Mgm't & Budget, Exec.

Office of the President, 0MB M-19-01, Request for Agency Feedback on the Federal Data Strategy, Federal Data Strategy, Principles 4-7 (201"9), available at https://www .whitehouse.gov/wp­ content/uploads/2018/1 0/M-19-0 l .pdf ("Conscious Design").

53 See id. Federal Data Strategy, Principle 2 ("Exercise Responsibility").

54 Section 302(a) of the Evidence Act requires statistical agencies and units to perform these functions through the application of policies, best practices and procedures, and requires agency heads to "enable, support, and facilitate" their performance of these responsibilities. 44 U.S.C. § 3563. Such policies, best practices, and procedures should also be promoted at the agency level.

55 Section IOl(a) of the Evidence Act requires the designated statistical official to "to advise on statistical policy, techniques, and procedures," and directs that "(a)gency officials engaged in statistical activities may consult with any such statistical official as necessary." 5 U .S.C. § 314( a). Section IO I ( a) further requires that "Each statistical official designated under subsection ( a) shall serve as a member of the Interagency Council on Statistical Policy established under section 3504(e)(8) of title 44." 5 U.S.C.

§ 314(b). 30

• Oversee: o Development of agency data quality standards to ensure that quality standards described in the Information Quality Act and Paperwork Reduction Act are met for statistical products produced by any part of the agency, including the provision of metadata to meet stakeholder needs and the permission to use privacy-enhancing technologies whose use requires significant amounts of documentation; and o Development of agency confidentiality standards to appropriately safeguard sensitive information contained in the agency's data assets.

The Statistical Official is expected to play a significant role in the agency's data governance activities, such as serving on the Data Governance Body. The Statistical Official is also expected to collaborate and consult regularly with other senior leaders such as the CDO 56 and Evaluation Officer on issues such as statistical methodologies, data access, data acquisition and linkage, protecting confidential data, and providing input to the learning agenda.

According to section lOl(a) of the Evidence Act, "(e)ach statistical official ... shall serve as a member of the lnteragency Council on Statistical Policy" (ICSP). 57 The ICSP, which is chaired by the Chief Statistician of the United States and coordinated within OMB's Office of Information and Regulatory Affairs (OIRA), will include the heads of each of the 13 principal.

statistical agencies, as well as the Statistical Official designated by the other CFO Act agency heads pursuant to the Evidence Act. 58 56 Section 202( e) of the Evidence Act specifies the functional relationship of the Statistical Official with the Chief Data Officer.

See 44 U.S.C. § 3520(d)(l) ("To the extent necessary to comply with statistical laws, the Chief Data Officer of an agency shall delegate any responsibility under subsection (c) to the head ofa statistical agency or unit (as defined in section 3561) within the agency."); id. § 3520(d)(2) ("To the extent permissible under law, the individual to whom a responsibility has been delegated under paragraph (I) shall consult with the Chief Data Officer of the agency in carrying out such responsibility."); id. § 3520(d)(3) ("The Chief Data Officer of the agency shall defer to the individual to whom a responsibility has been delegated under paragraph

(I) regarding the necessary delegation of such responsibility with respect to any data acquired, maintained, or disseminated by the agency under applicable statistical law.").

57 5 U.S.C.

§ 314(b). 58 See supra note 43 for a listing of the 13 principal statistical agencies. 31

APPENDIX D: FURTHER GUIDANCE ON ASSESSMENTS AND PLANNING

Annual Evaluation Plan I. Purpose The Evidence Act requires agencies to develop an Annual Evaluation Plan to be submitted in conjunction with the agency's Annual Performance Plan. These evaluation plans, per the Evidence Act, describe the evaluation activities the agency plans to conduct in the fiscal year following the year in which the performance plans are submitted and should include "significant" evaluations related to the learning agenda and any other "significant" evaluation, such as those required by statute. 59 The significance of an evaluation study should be defined by each agency and take into consideration factors such as the importance of a program or funding stream to the agency mission; the size of the program in terms of funding or people served; and the extent to which the study will fill an important knowledge gap regarding the program, population(s) served, or the issue(s) that the program was designed to address. Agencies should clearly state their criteria for designating evaluations as "significant" in their plans.

Evaluation plans offer agencies the opportunity to plan and document their approach to evaluation and, in particular, how their intended evaluations will support the agency's learning agenda. These plans will describe the systematic collection and analysis of information about the characteristics and outcomes of programs, projects, and processes as a basis for judgments, to improve effectiveness, and/or inform decision-makers about current and future activities. Annual Evaluation Plans support effective and efficient government by requiring agencies to think proactively and methodically about how they will use evaluation to help them operationalize and implement the activities needed to answer the agency's key questions as laid out in the learning agenda. Further, the publication of these plans supports more effective government as it allows the public to understand what an agency plans to evaluate and how it plans to do so. Publication also holds the agency accountable to undertake those activities. As evaluation is only one of many types of evidence that agencies will use to answer the priority questions identified in their learning agendas, the Annual Evaluation Plan is meant to help agencies chart their evaluation activities, and thus should reflect an agency's needs, priorities, and structure.

A number of agencies already use evaluation plans or comparable documents internally as a way to organize and monitor their evaluation activities. However, few agencies publish these plans to the extent required in Section 101 of the Evidence Act.60 The Evidence Act, for the first time, requires agencies to create these documents consistently and to publish them. Importantly, this requirement to publish Annual Evaluation Plan will foster greater accountability and transparency in Federal evaluation activities and, ideally, the results of those activities. Having a written evaluation plan will foster openness and encourage agreement with and support of the purpose, use, and users of the evaluation results. 0MB recognizes that each agency is different, 59 Per section 101 (a) of the Evidence Act, "[t]he term 'evaluation' means an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency." 5 U.S.C.

§ 311(3).

60 5 U.S.C.

§ 312(b). 32

and that there will be no single approach or structure for the Annual Evaluation Plan that will work for all agencies. II. Creating an Annual Evaluation Plan Requirements for an Annual Evaluation Plan The Evidence Act requires CFO Act agencies to issue an Annual Evaluation Plan concurrent with the Annual Performance Plan. Additionally, 0MB strongly encourages non-CFO Act agencies, as well as sub-agencies, operational divisions, bureaus within CFO Act agencies to develop and implement their own evaluation plans that tie into and are consistent with, but not limited by, the agency-wide plan as appropriate.

Management and Leadership of the Process Per the Evidence Act, the agency's designated Evaluation Officer shall lead, coordinate, develop, and implement the Annual Evaluation Plan. The Evaluation Officer shall play a leading technical role in the development and implementation of the Annual Evaluation Plan at the agency level and also support efforts to develop evaluation plans at the sub-agency, operational division, or bureau level, particularly as the activities in those plans relate to the agency-wide evaluation plan and learning agenda. Depending on the agency, other senior leaders may play an important role in creating the Annual Evaluation Plan. Agency leaders are also critical partners in developing these plans, as their support and buy-in to planned evaluation activities are essential for their success.

Stakeholder Engagement Agencies should consult with internal and external stakeholders as they develop and implement their initial Annual Evaluation Plan and those that will follow. This engagement will likely differ by agency depending on its needs, stakeholders, evaluation activities, etc. Internal consultation within the agency should, at a minimum, include those offices and staff that have a role in either undertaking evaluations or using their results. 0MB also encourages broader consultation with additional stakeholders to the extent feasible and appropriate. 0MB recognizes that extensive consultation on an annual basis may be challenging, and urges agencies to undertake processes to engage stakeholders as effectively and efficiently as possible. The process of stakeholder consultation, particularly within an agency, on evaluation plans supports collaboration and a sense of shared purpose, which increases the utility of evaluation findings to improve program effectiveness for the people and communities served.

Timing of Activities to be Included in the Plan The Annual Evaluation Plan will describe the evaluation activities the agency intends to conduct based on the learning agenda (discussed further in Appendix B) and other statutory mandates during the fiscal year following the year that the plan is submitted (i.e., for plans that outline evaluation activities for FY 2022, agencies will submit draft plans to 0MB in FY 2020 with 33

posting of final Annual Evaluation Plans in February 2021). Following the initial submission of the Annual Evaluation Plan, agencies can use future submissions to update and revise previously planned evaluation activities based on work conducted and new developments.

Content of the Annual Evaluation Plan Agencies are encouraged to create Annual Evaluation Plans that reflects the agency's needs and activities. The plan should not be a laundry list of agency evaluation activities. Rather, it should explain how, taken together, the planned evaluation activities help to answer the priority evaluation questions identified in the learning agenda, and either build upon or lay the foundation for other evidence-building activities: At a minimum, an agency's Annual Evaluation Plan shall describe the elements outlined below.

Questions to Be Answered. Annual Evaluation Plans shall describe the key questions to be addressed by each significant evaluation study or phase of an existing evaluation study that the agency plans to begin in the next fiscal year (i.e., the time period covered by the Annual Evaluation Plan).61 In discussing key questions, evaluation plans may necessarily need to describe a program's purpose, goals, and objectives, as well as discuss how program activities are linked to their intended effects. As appropriate, agencies may consider including program logic models in their evaluation plans to visually depict how a program being evaluated is expected to function and achieve its goals, specifying the program's inputs, activities, outputs, and outcomes. In this section, agencies should also discuss any evaluation activities that relate to its proposed regulatory actions in the Unified Agenda of Federal Regulatory and Deregulatory Actions, recognizing that these activities often need to occur well before the development of economically significant regulatory actions. To the extent that earlier evaluation activities influence the activities proposed in the evaluation plan, those should also be mentioned (e.g., revisions to evaluation questions based on a review of preliminary findings).

Information Needed for Evaluations. Annual Evaluation Plans shall describe the key information that will be used in the evaluations outlined in the plan. Specifically, agencies shall note whether they will undertake new information collections as part of the evaluation studies discussed or if they will acquire existing information for those evaluations.

Methods to Be Used. Annual Evaluation Plans shall describe the methods that agencies. intend to use for the evaluations. To the extent practicable, agencies shall articulate the evaluation design (e.g., experimental design or randomized controlled trial, quasi-experimental design, pre:.

post design, implementation study). As appropriate, agencies may also discuss any other analytic considerations, such as planned subgroups of interest. 0MB recognizes that providing this level of detail in advance of the actual initiation of evaluation activities may pose challenges for some agencies, particularly if they use independent, third-party organizations to conduct program evaluations and rely on those organizations to provide expert opinions on evaluation designs.

Nevertheless, it is OMB's expectation that in undertaking evaluations, agencies will use the most 61 The significance of an evaluation study should be defined by each agency and take into consideration factors such as the importance of a program or funding stream to the agency mission, the size of the program in terms of funding or people served, the extent to which the study will fill an important knowledge gap regarding the program, population(s) served, or the issue(s) that the program was designed to address. Agencies should clearly state their criteria for designating evaluations as "significant" in their plan.

34

appropriate method for the research question being asked. To that end, agencies may include methods that are most appropriate to answer the key questions included in evaluation plans, with the expectation that agencies will update any methods in future iterations of the Annual Evaluation Plan.

Anticipated Challenges. To the extent feasible and appropriate, agencies should discuss anticipated challenges posed by the evaluations included in the Annual Evaluation Plan. For instance, agencies may anticipate that following participants for several years to measure the long-term impacts of a program or intervention will be challenging, particularly if the population being studied is mobile. In this case, the agency might discuss ways to address sample attenuation. Conversely, agencies may encounter unexpected challenges that necessitate revisions to previously submitted evaluation plans.

Dissemination. Central to an agency's plans for evaluation activities is how the agency proposes to disseminate and use the results of those activities. The Evidence Act is clear that agencies should enhance their capacity to build and use evidence. Thus, agencies should think about and plan for how they will use evaluation results in making decisions about the programs, policies, and regulations of the agency before they begin evaluations and not as an afterthought. To the extent feasible, agencies' Annual Evaluation Plans should include discussion of how they intend to disseminate and use results for program, policy, and regulatory decision-making, in accordance with applicable law and policies governing the collection, use, and disclosure of data and information.

0MB recognizes that forecasting evaluation activities in advance, including the information described above, may be challenging. In their Annual Evaluation Plans, agencies should address the above categories of information in the level of detail that is feasible and appropriate given other considerations, such as availability of funding, procurement processes, data needs, and related matters. Agencies should be mindful of other laws relating to these areas, including the Antideficiency Act, the Federal Acquisition Regulation, agency-specific acquisition regulations, and any other applicable laws. .

As agency learning agendas are expected to include a range of methods and types of evidence (e.g., foundational fact finding, performance measurement, policy analysis, and program evaluation) that could be brought to bear to answer the priority questions identified, it is understood that evaluation will only be one of those methods and be relevant for answering only some of the identified questions. The Evidence Act requires agencies to provide information on significant evaluation activities planned. At a minimum, it is OMB's expectation that agencies will include program evaluation activities in their Annual Evaluation Plans. However, to the extent that agencies would like to provide context for their evaluation plan by describing other evidence-building activities they plan to undertake, they are encouraged to do so.

Capacity Assessment The requirement to carry out a capacity assessment applies to the 24 Federal agencies covered by th~ CFO Act. Non-CFO Act agencies, as well as sub-agencies, operational divisions, bureaus within CFO Act agencies are strongly encouraged to carry out a capacity assessment as 35

appropriate. The capacity assessment, per the Evidence Act, requires that agencies provide a list of existing activities (e.g., programs, initiatives, etc.) and operations (e.g., administrative and support tasks) that are being evaluated or analyzed.62 Led by the Evaluation Officer, it must also include a determination of the extent to which the agency's evaluations, research and analysis efforts, and related activities are appropriate in terms of:

• Supporting agency needs;

• Balancing agency needs for learning and management, performance and strategic management, interagency and private sector coordination, and oversight and accountability; and

• Using appropriate methodologies. Finally, the Capacity Assessment must determine current agency capacity for:

• Planning and implementing evaluation activities, disseminating best practices and findings, and incorporating employee views and feedback; and

• Carrying out capacity-building activities in order to use evaluation research and analysis approaches and data in the day-to-day operations. The Capacity Assessment will provide senior officials with information needed to fulfill the Evidence Act's intent to improve the agency's ability to support the development and use of evaluation, coordinate and increase technical expertise available for evaluation and related research activities within the agency, and improve the quality of evaluations and knowledge of evaluation methodology and standards.

Agencies are encouraged to conduct their Capacity Assessment using a format and process best suited to their agency context but which produces an understanding of their agency's strengths and weakness and promotes the planning for and development of effective capacity-building activities. The Capacity Assessment is expected to provide agencies with a baseline against which they can measure improvements to the coverage, quality, methods, effectiveness, and independence of their agency statistics, evaluation, research, and analyses. 0MB will provide additional information on the content of this assessment in future communications.

62 5 U.S.C.

§ 306(a)(8) 36

APPENDIX E: IMPLEMENTATION RESOURCES FOR AGENCIES

0MB recognizes that agencies may need additional resources and support to implement the Evidence Act requirements addressed in this guidance. Below are several resources that agencies are encouraged to consult and use as they implement these requirements. Designated Official Orientation and Ongoing Training As noted in the Memo, all designated officials (i.e., Chief Data Officers, Evaluation Officers, and Statistical Officials) must attend an Evidence Act orientation hosted by 0MB in September 2019 focused on how to undertake and operationalize these positions. It is OMB's expectation that this orientation will provide concrete tools for these designees, as well as an opportunity to build a network for peer learning.

Federal Data Strategy Tools The Federal Data Strategy is currently developing a series of tools that agencies can use to support a range of data-related activities. As they become available, these tools will be available at https://resources.data.gov. Evidence and Evaluation Community of Practice The 0MB Evidence and Evaluation Community page on MAX (available at https://community.max.gov/x/iA OJQ) includes a curated library of materials to support agencies in developing learning agendas, writing annual evaluation plans, conducting capacity assessments, and general program evaluation activities, including:

• Example agency documents (e.g., learning agendas, evaluation plans, assessments);

• Toolkits and other reference tools;

• Training opportunities; and

• Other materials related to Evidence Act implementation. 37