← Back to research report

Document · 2019-04-24

M-19-15, Improving Implementation of the Information Quality Act

Complete document.

About this text

This text has been formatted for reading on screen. Page links open the original PDF. Older scans may contain errors in words, tables, or reading order.

Document sections
Jump to a PDF page · 11 included

E,XECUTIVE OFFICE OF THE PRESIDENT OFFICE OF MANAGEMENT AND BUDGET WASHINGTON, D.C. 20503

April 24, 2019 M-19-15 MEMORANDUM FOR THE HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES FROM: Russell T. V oughf \jActing Director

SUBJECT:

Improving Implementation of the Information Quality Act

Introduction

The purpose of this Memorandum is to reinforce, clarify, and interpret agency responsibilities with regard to responsibilities under the Information Quality Act (IQA).1 In 2002, the Office ofManagement and Budget issued Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility and Integrity ofInformation Disseminated by Federal Agencies2 ("Guidelines"). The principles and core responsibilities described in the Guidelines remain sound and relevant for agency practice; however, additional guidance is required to address changes in the information landscape and to incorporate best practices developed over time. 3 This Memorandum updates implementation of the Guidelines to reflect recent innovations in information generation, access, management, and use, and to help agencies address common problems with maintaining information quality.

Background

Prudent decision making depends on reliable, high-quality information. Congress has long recognized that federal agencies should make decisions using the best data reasonably available, and Congress has entrusted 0MB with the statutory role of ensuring that federal

1 Treasury and General Government Appropriations Act, 2001, Pub.

L. No. 106-554, § 515(a) (2000) (as codified at

44 U.S.C.

§ 3516, note).

2 Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity oflnformation

Disseminated by Federal Agencies, 67 FR 8452 (Feb. 22, 2002), available at https:/ /www.federalregister.gov/ documents/2002/02/22/R2-59/ ....uidelines-for-ensurin b -and-maxim izing-the-qual it - ob·ectivit -utilit -and-inte rit -of-information.

3 Id.

at 845 8 ( contemplating the evolution of standards with experience and the need for additional implementation guidance); cf Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility and Integrity oflnformation Disseminated by Federal Agencies, 66 FR 49,718, 49723 (September 28, 2001); John D. Graham, Memorandum for the President's Management Council (October 4, 2002), available at https:/Iobamawhitehouse. archives.gov/ sites/ default/files/ omb/ assets/ omb/inforeg/pmc _graham_ 100402.pdf.

agencies collect, use, and disseminate information that is fit for its intended purpose. 4 Within 0MB, the Office of Information and Regulatory Affairs (OIRA) works with agencies to maintain information quality standards. Implementing statutory requirements in the IQA, th~ Guidelines provide a framework for oversight ofthe quality of information disseminated by the federal government5 throughout its lifecycle, which inclµdes creation, collection, pre-dissemination review, transparent and reproducible use, and ultimately correction and disposition. 6 The Guidelines impose three core responsibilities· on federal agencies:

1. Agencies must embrace a basic standard of quality and consider quality in their information dissemination practices.

2. Agencies must develop information quality assurance procedures that are applied before dissemi11:ating inf01mation.

3. Agencies must develop an administrative mechanism for affected parties to request that agencies correct information of inadeq1,1ate quality, with an appeal process and annual reports to 0MB. In response-to the Guidelines, agencies prepared agency-specific guidelines setting forth procedures to ensure information quality. This Memorandum directs agencies to update their guidelines within 90 days along-the following parameters.

1. Information:

Fitness for Purpose and Pre-Dissemination Review The IQA requires agencies conduct pre-dissemination review oftheir information products.7 During this review, each agency should consider the appropriate level of quality for each ofthe products that it di~seminates based on the likely use ofthat information. The Guidelines explain that quality encompasses utility, integrity, and objectivity. Utility refers to the 4 See, for instance, the Federal Reports Act of 1942; the Budget ang Accounting Procedures Act of 1950; the Paperwork Reduction Act of 1980 and its 1995-amendments;_ the Information Quality Act; and the Foundations for Evidence-Based Policymaking Act of2018, Pub. L. No. 115-435, 132 Stat. 5529 (2019). _ .

5 "Dissemination" means agency initiated or sponsored distribution of information to the public. See 5 C.F.R. 1320.3(d) (defming "Conduct or Sponsor"). Dissemination does not include distribution limite_d to government employees or agency contractors or grantees; intra-or inter-agency use or sharing of government information; and responses to requests for agency records under the Freedom oflnformation Act, the Privacy Act, the Federal Advisory_ Committee Act or other similar law. This defmition also does not include distribution limited to correspondence with individuals or persons, press releases, archival records, public filings, subpoenas or adjudicative processes. See Guidelines, 67 FR at 8460.

_6 Other guidelines implementing the IQA include the Final Bulletin for Agency Good Guidance Practices, 72 FR 3432 (2007). ··_ 7 Guidelines, sec. III.2, 67 FR at 8459 ("As a matter of effective agency infor111:ation resources management, agencies shall develop a process. for reviewing the quality (including the objectivity, utility, and integrity) of information before it is disseminated. Agencies shall treat infonn,tion quality as integral to every step of an agency's development ofinformation, including creation, collection, maintenance, and dissemination. This process shall enable the agency to subst~ntiate the quality ofthe information it has disseminated through 4ocumentation or other means appropriate to the information.").

2

data's utility for its intended us·ers and for its intended purpose. Integrity refers to th_e data's security.· Objectivity refers to whether the disseminated information is accurate, reliable, ancl unbiased as a ~atter ofpresentation and substance. The focus on the information's usefulness is critical; the. Guidelines recogpize that "information quality comes at a cost, "8 and "that some government infonn,ation may need to meet higher or more specific quality standards than those that would apply to other types of government information, "9 depending on the information's expected use. The touchstone is "fitness for purpose"; infornfation destined for a higher-impact purpose must be held to higher standards of quality. 10 The Guidelines characterize a subset of agency information as "influential scientific, financial, or statistical information" that is held to higher quality standards. This is scientific, financial, or statistical information that "the agency can reasonably determine ... will have or does have a clear and substantial impact on important public policies or important private sector decisions." For instance, the Principal Federal Economic Indicators11 are an example of influential statistical informatton. In the context of a policy d.ecision, a specific piece or body of information is "influential" when it is a principal basis for a decision by a federal decision­ maker, that is, ifthe same decision would be difficult to reach in t~at information's absence or-if the decision would lose its fundamental scientific, financial, or statistical underpinnings absent the information. Even if a decision is very important, a particular piece of information supporting it may or may not qe "influential," depending on whether the decision could be reached in the information's absence. Each agency is authorized to ·define whether information is "influential" given the nature of issues for which the agency is responsible.

Identifying "Influential" Information Implementation Update 1.1: Drawing on experience implementing the Guidelines, .agencies should revisit the parameters for identifying "influential information. "Agencies should provide specific guidance to program managers for determining the amount and type ofpre-dissemination review necessary. Agencies should identify specific types of information the agency produces that are "influential" and should provide a rigorous processfor determining whether types ofinformation not specifically listed by the guidelines qualify as "influential."

The Guidelines explicitly gave agencies the discretion to define "influential" in the context oftheir specific missions. Rather than tailor the definition, many agencies simply . adopted 0MB 's statement that "the agency can reasonably determine ... [ what information] will have or does have a clear and substantial impact on important public policies or important private sector decisions." Age.t;i.cies should provide greater specificity to enable program 8 Id at 8453.

91d at 8452. 10 Id at 8453. 11 The Principal Federal Economic Indicators (PFEis) are the major statistical ~eries that describe the current condition ofthe economy; examples include the unemployment rate, the Gross Domestic Product, and agricultural · prices and crop production. Some aspects of quality assurance for PFEis are articulated in Statistical Policy Directive on Compilation, Release, and Evaluation ofPrincipal Federal Economic Indicators, 50 FR 38,932 (Office ofM_gmt. & Budget Sept. 25, 1985). · 3

managers to determine if information is "influential" early in the information generation process and to impose suitable control measures. Peer Review ofInfluential Scientific Information Implementation Update 1.2: When using scientific information, including third-party data or models, to support their policies, agencies must ensure compliance with the requirements ofOMB's Information Quality Bulletin for Peer Review.

Implementation Update 1.3: When conducting peer review, agencies should ensure reviewers are asked to evaluate the objectivity ofthe underlying 4ata and the sensitivity ofthe agency's conclusions to analytic assumptions. Implementatio11: Update 1.4: When influential information that has been peer reviewed changes significantly (e.g., as a result ofthe peer reviewer comments, additional (J,gency analysis, or further consideration), the agency should conduct a second peer review.

The Guidelines emphasize the importance ofpeer review as a tool for determining fitness of scientific information for policy purposes. For the subset of scientific information that is "influential," peer review is a required component ofpre-dissemination review, as described in OMB's Final Information Quality Bulletin for Peer Review (Bulletin). 12 The Bulletin's.purpose is to increase the quality and credibility of scientific information used by the government. The Bulletin includes requirements for the selection ofreviewers and transparency ofthe review, as well as guidance c;m selecting the appropriate mechanism for peer review and the importance of providing explicit instructions to reviewers (i.e., a peer review charge). Even after the 2004 Bulletin, only some agencies have robust peer review mechanisms.

Proper peer review includes, inter alia, that .agencies peer review complex models underlying economically significant regulations before submitting those draft regulations to OIRA under Executive Order 12866. Moreover, agencies must send OIRA the required annual rep(?rtS of completed peer reviews of influential scientific information. These reports are essential for OIRA to monitor agency compliance with the standards ofthe IQ.~:

2. Downstream.

Use of Data Below we discuss additional data access policies promulgated by 0MB to both increase · taxpayer return on federal investment and to spur private sector innovation. Furthermore, there is a growing consensus· within both government and the private sector that emphasizes the importance of secondary use of data-analyzing data for a purpose other than the primary one . for which it ·was collected. These federal data access policies, in conjunction with responsibilities under the Guidelines, require agencies to ensure they have clear policies for evaluating and communicating the fitness-for-purpose of data made available to the public.

12 Office ofMgm't & Budget, Exec.

Office ofthe President, 0MB M-05-03, Memorandum for the Heads of Executive Departments and Agenctes: Issuance of 0MB 's "Final Information Quality Bulletin for Peer Review" (2004), available at https://www.whitehouse.gov/wp-content/uploads/2017/11/2005-M-05-03-Issuance-of-OMBs­ Final-Information-Quality-B ulletin-for-Peer-Review-December-16-2004.pdf.

4

Public Access to Government Information (Open Data) Implementation Update il: When an agency makes information originally collected or developed by other Federal agencies available to the public in a cross-agency dissemination, each agency is responsible for the quality ofthe information they contribute, and that responsibility should be clearly communicated to the public.

Implementation Update 2.2: Agencies should provide the public with sufficient documentation about each dataset released to allow data users to determine the fitness of the data for the purpose for which thirdparties may consider using it. Robust practices may include developing a standard template or framework that provides data users with the relevant information. Safeguarding privacy and confidentiality is vital in the context ofopen data.

The Open, Public, Electronic, anq, Necessary Government Data Act; 13 0MB Circular No. A-130 Managing Information as a Strategic Resource;14 and 0MB Memorandum M-13-13: Open Data Policy-Managing Information as an Asset15 require agencies to collect and create information in a way that supports :public transparency as well as downstream, secondary information dissemination and processing by third pa1iies, thereby making government information accessible, discoverable, and usable. Third parties include entrepreneurs; state, local, 'and tribal governments, scientists, private sector stakeholders, and non-governmental advocacy organizations. With respect to transparency, the Federal government has developed a number of cross-agency products (e.g., dashboards, data visualizations, and scorecards) through which information from various agencies is made available to the public in a consolidated manner. In these dissemination contexts, as in the publication ofjoint reports, each agency is responsible for the quality ofthe information that they contribute to the cross-agency product. 0MB policy emphasizes that, when data are made available to the public, potential users must be provided with sufficient information to understand which agency is responsible for the quality ofthe data being disseminated, as 'Yell as the data's strengths, weaknesses, analytical limitations, security req1:1irements,_ and processing options.16 · Additionally, 0MB policy requires agencies to ensure that privacy and confidentiality are .

fully protected and that data are properly secured so that open data do not disclose personally identifiable information. Agencies must account for the 'mosaic effect' of data aggregation, which occurs when the information in an individual dataset, in isolation, may not pose a risk of identifying an individual ( or t_hreatening some other important interest such as .security), but ·13 The Open, Public, Electronic, and Necessary (OPEN) Government Data Act, Pub. L. No. 115-435, 132 Stat. 5534

(2019) (Title II ofthe Foundatio!ls for Evidence-Based Policymaking Act of2018).

14 Office ofMgm't & Budget, Exec.

Office ofthe President, 0MB Circular No. A-130, Managing Information as a Strategic Resource (2013), available at https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/_circulars/Al30/a130revised.pdf.

15 Office ofMgm't & Budget, Exec.

Office ofthe President, 0MB M-13-13, Memorandum for the Heads of Executive Departments and Agencies: Open Data PolicY:-Managing Information as an Asset (2013), av.ailable at https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/2013/m-13-13.pdf. 16.See id at 4 ( defining open data, third principle; importance ofrobust meta data and preference for the use ofnon­ proprietary models to facilitate transparency and secondary use by the public).

5

when combined with other available information, could pose such risk. The mosaic effect demands a risk-based analysis, often utilizing statistical methods whose parameters can change over time. The federal government maintains various resources for assisting agencies with this analysis. For example, the Federal Committee on Statistical Methodology periodically updates guidance for agencies on applying statistical disdosure limitation methods· to publicly available datasets and provides technical assistance on .applying these methods. 17 Re-use ofExisting Agency Program Data Implementation Update 2.3: Agencies should con.sider the potential for using existing data sources from both inside and outside the agency for statistical and res,earch purposes, while protecting privacy and confidentiality.

Implementation Update 2.4: When designing or improving data collection systems, Departments should actively solicit comment from their statistical, research, and evaluation agencies about potential downstream uses. Agencies should describe such uses in the Information Collection Request submitted to 0MB for review under the P RA.

Implementation Update 2.5: Ifagencies are considering secondary analysis ofdata that includes personally identifiable information, the agencies should coordinate with their Senior Agency Official for Privacy to meet all privacy requirements and manage privacy 4 risks.

Implementation Update 2. 6: Agencies should develop procedures for clearly documenting an·d communicating the quality ofadministrative data that have the potential to be usedfor statistical purposes.18 0MB Memorandum M-14-06: "Guidance for Providing and Using Administrative Data for Statistical Purposes,"19 encourages agencies to create statistical information more efficiently . through greater use ofinformation.that the Federal Government has already collected for programmatic, regulatory, or administrative purposes.2°For instance, secondary analysis of administrative data may be used to eva~uate how 'Yell the policy·or program that collected the 17 Fed. Comm. on Stat. Methodology, Statistical Policy Working Paper 22: Report on Statistical Disclosure Limitation Methodology (2d version 2005), available at https://nces.ed.gov/FCSM/pdf/spwp22.pdf.

18 Per 0MB M-14-06, "statistical purposes" would include use ofthe data for research and program evaluation so long as the analytic approach is to describe, estimate, or otherwise analyze the characteristics or experience of groups, "without identifying the individuals or organi~ations that comprise such groups." Office ofMgmt. & Budget, Exec. Office ofthe President, 0MB M-14-06, Memorandum for-the Heads ofExecutive Departments and Agencies: Guidance for Providing and Using Administrative Data for Statistical Purposes 4 N.2 (2014) (citing Confidential Information Protection and Statistical Efficiency Act of2002, Pub. L. No. 107-347, §502(9)(A)), available at https://www.whitehouse.gov/sites/whitehouse. gov /files/omb/memoranda/2013/m-14-06.pdf.

19 Id.

20 "Administrative data," for purposes ofM-14:-06 and this Memorandum, "refers to administrative, regulatory, law enforcement, adjudicatory, financial, or other data held by agencies and offices ofthe government or their contractors or grantees (including States or other units ofgovernment) and collected for other than statistical purposes. Administrative data are typically c·ollected to carry out the basic administration of a program, such as processing benefit applications or tracking services received. These data relate to individuals, businesses, and other institutions.'.' Id. at 4 n. l.

6

information is working. This practice, which must be conducted in a manner that respects _privacy and_confidentiality, increases the value ofthe existing data, reduces costs associated with collecting new data, and reduces burden on respondents. The bipartisan Commission on Evidence-Based Policymaking21 and the National . Academies of Sciences, Engineering, and Medicine (the National Academies)22 have called for increased use of existing Executive Branch program data for evaluating the effectiveness of government programs and policies. To increase the integrity _of analyses based on such data, the National Academies recommends a comprehensive quality framework that includes evaluating and documenting the timeliness, relevance, accuracy, accessibility, coherence, integrity, privacy, transparency and interpretability, and granularity of each data source used. Reports by both bodies emphasize the importance ofprotecting personal information and discuss approaches for maintaining privacy and confidentiality while increasing access for statistical purposes.23 The Foundations for Evidence-Based Policymaking Act of 2018 codifies these concepts as agency responsibilities. Among other things, agencies must develop evidence-building plans, identify the data needed to build evidence, and unless expressly prohibited by law, make their data available upon request to any statistical agency o~ unit for purposes ofdeveloping evidence. 24

3. Reproducibility of Influential Information

The Guidelines include a "reproduciqility standard" for influential information. The purpose ofthe reproducibility standard is to increase the credibility of federal decisions. The standard requires that influential analys(?S must be disseminated with sufficient descriptions of data and methods to allow them to be reproduced by qualified third parties who may want to test the sensitivity of agency analyses. This is a higher standard than simply documenting the characteristics ofthe underlying data, which is required for all information ( as discussed in the previous section).

Models and Machine Learning Implementation Update 3.1: Consistent with the Office ofScience and Technology Policy's 2010 Memorandum for the Heads of Executive Departments andAgencies: Scientific Integrity, agencies should ensure that influential information is communicated transparently'by "including a clear explication ofunderlying assumptions,· accurate contextualization ofuncertainties,· and:a description ofthe probabilities associated with · 21 Comm'n on Evid.-Based Policymaking, The Promise ofEvidence-Based-Policymaking (2017), available at htfps"://www.cep.gov/cep-final-report.html.

22 Nat'l Acads.

of Sci., Eng'g, and Med., Innovations in Federal Statistics: Combining Data Sources While Protecting Privacy (2017), available at https://www.nap.edu/catalog/24893/federal-statistics-multiple-data-sources­ and-privacy-protecti6n-next-steps. 23 The National Academies and the Commission reports adopt the definition ofstatistical purposes used in 0MB M- 14-06. When blending survey and administrative data, the National Academies calls for a more comprehensive quality framework to include dimensions that capture interoperability concerns, including timeliness, relevance, accuracy, accessibility, coherence, integrity, privacy, transparency and interpretability, and granularity.

24 Pub.

L. No. 115-435. 7

both optimistic and pessimistic'projections, including best-case and worst-case scenarios. "25 Implementation Update 3.2: When an agency has performed analysis using a specialized set ofcomputer code, the computer code used to process it should be made available to the public for further analysis, ifconsistent with applicable law and policy: Often influential information includes both an analysis and the underlying data, and both should be made available to the public absent contrary concerns such as privacy. In the context of results generated by, for instance, a statistical model or machine· augmented learning and decision support,26 reproducibility requires, at minimum, transparency about the specific methods, design parameters, equations or algorithms, parameters, and assumptions used.

Non-Government Information Implementatio_n Update 3.3: Agencies should ensure that when using non-government . sources to create influential information they communicate to the public sufficient information on the characteristics ofthe data and analysis, including its scope (e.g., .

temporal or demographic), generation protocols, and any other information necessary to allow-the public to reproduce the-agencies' conclusions. Non-government information such as data or a model may be influential in a government (! decision. The Guidelines apply to "information that is not created by the Federal Government when agency use ofthat information provides the appearance ofrepresenting agency views (e.g., being relied upon to support a·rulemaking, risk assessment, or agency policy)." Examples of · non-government information that federal agencies have used as the basis ofimportant public policy decisions include scientific research published in peer-reviewed journals, data submitted by industry or non-governmental organizations in response to agency Requests for Information, and information generated by state, local, tribal, or international governments. Newer sources of non-government party information include data generated from web scraping exercises, data purchased from the private sector ( e.g., credit-card transactions or utility company records of water or energy use), and data generated by sensors and satellites.

Access to and Considerations for Protecting Data Implementation Update 3.4: Agencies should prioritize increased access to the data and analytic frameworks (e.g., .models) used to generate influential information. All data disclosures must be consistent with statutory, regulatory, and policy requirements for protections ofprivacy and confidentiality, proprietary data, and confidential business information.

25 John P.

Holdren, Office of Sci. & Tech. Pol'y, Memo:i;andum for the Heads of Exec1=1tive Departments and Agencies: Scientific Integrity (2010), available at https:/Iobamawhitehouse.archives.gov/sites/ default/files/microsites/ ostp/scientific-integ~ity-memo-121720 IO.pdf.

· 26 Currently, structured and unstructured machine learning are common forms of artificial intelligence used in a policy context. The Guidelines, however, apply to other current and future forms of artificial intelligence ifthey are used to create information disseminated by the Federal government.

8

Implementation Update 3.S: Agencies should explore methods that provide wider access to datasets while reducing the risk ofdisclosure ofpersonally,identifiable information. In particular, tiered access offers promising ways to make data widely available while protecting privacy. Implementation ofsuch approaches must be consistent with principles for ethical governance, which include employing sound data security practices, protecting individual privacy, maintaining promised coefidentiality, and ensuring appropriate access and use.

To ensure reproducibility, the Guidelines set an expectation of access to data underlying influential information, subject to "compelling interests such as privaoy, trade secrets, intellectual property, and other confidentiality·protections."27 Since the 2002 GZJidelines, the technology for allowing protected access to data has progressed stgnificantly'. New approaches to secure data access using cutting-edge technologies reduce the risk of re-identification and therefore may mitigate certain privacy risks associated with providing such access. Risk reduction techniques include creating multiple version_s of a single dataset with varying levels of specificity and protection (sometimes referred to a "tiered acc~ss").28 The virtue oftiered access is that data users who wish to conduct activities with a statistical purpose without first obtaining special authorization have access to the versions ofthe data in the least restricted tiers, allowing them to conduct research while protecting confidentiality. Such approaches to increasing access to data for statistical purposes could be considered by more federal agencies, thereby allowing stakeholders to replicate analyses and explore the sensitivity of the conclusions to alternative assumptions while accessing only the data th~y need. As agencies consider adding "intermediate tiers between fully open and fully closed, they must build in sufficient controls to monitor who is accessing the data and allow access only for authorized purposes.29

4. Requests for Correction

The Guidelines require agencies to establish administrative mechanisms to allow the public to submit a "request for correction" (RFC) when disseminated information does not comply with agency guidelines, as well as an opportunity request reconsideration ofthe agency's initial decision qn a RFC.

27 Guidelines, sec. V(3)(b)(ii)(B), 67 FR at 8460. 28 Access to the most restricted versions is limited to authorized researchers for approved statistical and research purposes. Such access often takes place in secure physical or virtual enclaves. To maintain confidentiality, less restricted versions typically sacrifice specificity or granularity in exchange for easier access. See, e.g., Comm 'non Evid-Based Policymaking, supra note 21, at 23-46. Data query tools run analyses without ever revealing individual records. The least restricted, and least specific, data released by federal agencies are often public use files, which mask individual records in order to maintain confidentiality or aggregate data such as in tabular form. To reduce the risk ofre-identification, agencies apply statistical disclosure limitation methods, such as data swapping or recodii?-g, to publicly available datasets. See, e.g., Fed. Comm. on Stat. Methodology, supra note 17.

29 See, e.g., id 9

·Processing Timelines Implementation Update 4.1: Agencies should revise their procedures to reflect more realistic timelines for RFCs. Revisedprocedures should, at minimum, provide that agencies will not take more than 120 days to respond to an R,FC without the concurrence ofthe party that requested the request for correction.

Agencies freqµently unilaterally extend their own deadlines for" replying to RFC, taking a year or more to provide a substantive response. Excessive response times do not allow for correction in a timely manner. Agencies should set and adhere to reasonable timelines, not to exceed 120 days, for a response without the concurrence ofthe requester.

Sharing Draft Responses with 0MB Prior to Release Implementation Update 4.2: In its response to an RFC, agencies should not opine on the requestor 's or the agency's policy position. Implementation_ Update 4.3: The agency response should contain a point-by-point response to any data quality arg11ments contained in the RFC and should refer to a peer review that directly considered the issue being raised, ifavailable.

Implementation Update 4.4: Agencies should share .draft responses to RFCs and appeals with O MB prior to release to the requestor for assessment ofcompliance with the above norms. The process described here is designed to challenge technical information that drives policy, not to debate the policy itself. At times, agencies have failed to respond fully to the.

technical challenges posed by requestors. Under the IQA and Guidelines, an agency should respond thoroughly to substantive RFCs, including by making clear, as fully as practicable, the data underlying the challenged information, the methodologies the agency used to analyze the data, the reasons for use of such methodologies, and any peer reviews addressing the agency's analysis. .

Appeals Requests Implementation Update 4.5: To ensure the integrity ofthe appeals process, agencies should ensure that those individuals reviewing and responding to the appeals request were not involved in the review and initial response to the RFC Age~cies usually forward RFCs and appeals requests to the office that released the information at issue for consideration, deliberation, and genetation·ofthe draft response. -While that practice .is permissible, the same individuals who opine during the initial response should not participate· in the appeals process. Because the RFC process described here should focus on interpretation oftechnical/scientific information, at both the initial and appeals request level, the staffreviewing any appeals should be versed in the process of interpreting the type of evidence in question ( e.g., financial, scientific·, or statistical information). Agencies. should have 10

procedures in place to ensure that the .staff reviewing appeals is independent ofthe staff who prepared the initial response, and forms an independent judgment ofthe RFC. Likewise, staff reviewing appeals should be sufficiently senior that they are effectively able to disagree with the assessment of colleagues who prepared ~he initial response. Some agencies have established a panel of senior managers to review appeals. A panel of senior managers can be an effective mechanism to ensure that the issues raised in the appeals are given independent consideration by competent and fair-minded decision-makers.

11