Blog Posts in Peer Review

EPA Scientific Integrity Proposal Missing Critical Elements

 

The U.S. Environmental Protection Agency’s (EPA) draft scientific integrity policy is missing critical elements needed to effectively safeguard science at the agency, OMB Watch said in comments filed yesterday. The policy must be improved if the agency is to ensure that the best science informs policy decisions that affect the health and environmental quality of all Americans.

The Obama administration recognizes that sound, uncensored science is critically important to protecting our health, economy, and environment. Consequently, President Obama issued a memo shortly after taking office, establishing protections for scientific integrity and directing agencies to implement them. Unfortunately, implementation of that memo has been sluggish and uneven. However, there have been some bright spots, such as the draft policy at the National Oceanic and Atmospheric Administration (NOAA), which OMB Watch praised.

EPA's efforts have been more mixed. For example, the process to develop a policy at EPA has been mostly open: the agency posted a draft policy and solicited public comment. But the openness has been less than full-fledged, as important appendices and references are missing from the policy posted. EPA should be sure the public has an opportunity to comment on a complete version of the policy before finalizing it.

The content of EPA's draft policy is also flawed. The draft policy moves in the right direction but stops short of what's needed to adequately protect scientific integrity. Fundamentally, an effective scientific integrity policy must do two things: prevent political interference with science and protect the free flow of scientific information. EPA's draft policy is inadequate on both counts. As we say in our comments:

In general, the portions of EPA’s draft policy that have been released to date establish the appropriate principles for scientific integrity, particularly in striving to keep science free from political interference and to foster a culture of scientific openness. However, the translation of these principles into effective policies is lacking, and we recommend that EPA make significant changes to the draft policy to address this gap.

To improve EPA's draft policy, OMB Watch makes these recommendations in our comments:

  1. Make the prohibitions on political interference with science enforceable;
  2. Strengthen protections for the free flow of scientific information;
  3. Protect personnel who blow the whistle on scientific integrity violations;
  4. Improve scientific integrity in peer review and federal advisory committees;
  5. Expand the role and responsibilities of the Scientific Integrity Committee; and
  6. Strengthen scientific integrity in interagency processes.

EPA's draft suggests that the agency understands the central role of scientific integrity in achieving its critical work to protect Americans' health and natural resources. But EPA needs more than just principles in order to have a policy that effectively upholds scientific integrity. Luckily, EPA has a great model to draw from: the draft NOAA policy. We're hopeful that EPA will make the revisions necessary to ensure that EPA science continues to be top-caliber research that Americans can trust.

Follow Gavin Baker on Twitter

(Gavin Baker 09/07/11; 2 comments)

Three Months Later, Missing "Oil Budget" Methodology Released

 

The government responders to the BP Deepwater Horizon oil spill today released its report on the "oil budget" tool used to calculate what happened to the spilled oil. Unlike the version of the report released on Aug. 4, this one comes peer-reviewed and with methodology.

As we noted in our earlier commentary, the August report was criticized for not disclosing the methods used to reach its conclusions. As a congressional staffer wrote at the time, "Essentially, nobody can check the math."

Critics also noted the administration's claims that the August report had been peer-reviewed by independent scientists turned out to be untrue. National Oceanic and Atmospheric Administration (NOAA) administrator Jane Lubchenco acknowledged today, "That report was not peer-reviewed, and I was in error." The new report contains an appendix listing 14 peer reviewers and their responses to an earlier draft of the document – a welcome step to provide much-needed transparency.

Perversely, the peer review process delayed the disclosure of the study's methodology but not the conclusions. Chairman Ed Markey (D-MA), in an Aug. 19 hearing of the House Energy and Environment Subcommittee, asked NOAA's Bill Lehr if the methodology would be released. Lehr replied that it would be – but only after completing a peer review process that could take several more months. The report released today is the result of that process.

Peer review is an important part of the scientific process, but it was a poor reason to withhold the report's methodology. If the administration was confident enough in the study's conclusions to release them publicly, then it should have been confident enough to release the methodology, too.

Quality control does take time, and there can be a value in releasing earlier drafts – particularly in a situation like the BP spill, where there was tremendous demand for timely information. But the tension between quality and promptness cannot be resolved by publicly trumpeting conclusions while their very basis is still under review and hidden from the public.

Science isn't based on assertion: it's based on disclosed methods that others can examine and critique. The public shouldn't have to take the government at its word. The administration should seriously consider the Oil Spill Commission's suggestion that government study groups release more of their underlying methodology and data.

(Gavin Baker 11/23/10; 0 comments)

What Happened to Obama’s Commitment to Scientific Integrity?

 

Today, March 9, is the one-year anniversary of President Obama’s scientific integrity memo which instructed his staff to produce within 120 days recommendations for ensuring independence of federal scientists and limiting political interference in their work. 365 days later, we’re still waiting.

expect delays The Union of Concerned Scientists is critical of the delay. Francesca Grifo, director of UCS’s scientific integrity program, had this to say:

While the new administration has been generally supportive of scientific integrity values, it's moving too slowly to establish badly needed reforms. The current system still discourages scientists from communicating about their research results, for example. It still keeps the public in the dark about the scientific basis for policy decisions, and it still rewards staffers who keep quiet about political interference in science. 

The criteria Obama laid out in his March 9, 2009, memo are admirable: hiring and keeping qualified scientists; defining new policies to ensure integrity; using “well-established scientific processes” like peer review; disclosing scientific findings; ensuring that scientific integrity principles are being adhered to; and adopting additional policies like whistleblower protections.

But by failing to follow up with a concrete set of reforms, Obama and John Holdren – the Director of the Office of Science and Technology Policy who was tasked with developing recommendations – are sending a terrible message to those who believe scientific integrity ought to be a priority for this administration.

Interference in science reached new heights under President George W. Bush; but just because Bush is gone does not mean the problems go away too. As OMB Watch discusses in the latest issue of our e-newsletter The Watcher, a new report from the Project on Scientific Knowledge and Public Policy (SKAPP) proves that much work remains. SKAPP interviewed federal scientists during both the Bush and Obama administrations, and found that although there were a few bright spots in scientists' views of the changes that had occurred, a majority felt similar frustrations.

Couple this delay with the now year-plus delay on Obama’s effort to improve the regulatory process by writing a new executive order, and my outlook on the administration’s commitment to government reform is dimming.

Photo by Flickr user davidfntau. Used under a Creative Commons license.

(Matthew Madia 03/09/10; 2 comments)

EPA Releases OMB Comments on Chemical Studies

 

As promised, the Environmental Protection Agency is releasing White House comments on EPA assessments of potentially toxic chemicals. Yesterday, EPA released the first batch of comments on four ongoing risk assessments.

OEOBIn May, EPA announced changes to its Integrated Risk Information System (IRIS), a program that studies the health effects of chemicals and releases the findings on the EPA website. Under the new process, there are two opportunities for “interagency” review – that is, a chance for other government agencies, including the White House, to scrutinize and comment on EPA’s risk assessments.

The four comments released yesterday all came from the White House Office of Management and Budget. (Under the old process, OMB reviewed assessments, sometimes with detrimental results.) From my initial reading, they appear to abide by EPA’s stipulation that agencies only comment on the relevant scientific issues and refrain from injecting broader policy or political considerations.

The revised IRIS process required that interagency comments be posted on the EPA website. They can all be found here, www.epa.gov/ncea/iris/index.html, in the table under “New Assessments and Reviews,” then click on “Downloads and Related Links” on the next page. (For a fifth assessment, cerium oxide and cerium compounds, no comments are listed. I’m not sure if this is an oversight, or if no comments were filed.)

The whole idea of OMB review of IRIS assessments continues to give me pause. OMB is not a scientific agency, and it is unclear what value OMB officials add to IRIS. As I said, OMB and other agencies get two opportunities to review the IRIS assessments: once after EPA prepares an initial draft, and again just before a final version is published (after the assessment undergoes an external peer review).

What good does it do for an office in the White House to review a scientific study after it has been peer reviewed? Three of the four assessments on which OMB has commented thus far are in the second interagency review phase. For two of the assessments, OMB complements EPA for being “very responsive” to the peer review. But in the third, OMB included four pages of scientific concerns.

When EPA revised the IRIS process in May, EPA Administrator Lisa Jackson said, “The process will be entirely managed by EPA.” That was a critically important reform, because EPA should now be able to keep OMB in check. As long as EPA stays in control, there shouldn’t be any problems, but we should still keep a watchful eye on any OMB comments.

(Matthew Madia 10/15/09; 1 comment)

Questions Unanswered on White House Role in EPA IRIS Process

 

As I blogged a couple weeks ago, the Environmental Protection Agency is attempting to streamline its process for assessing the health effects of chemicals under its Integrated Risk Information System (IRIS).

The new process removes some unnecessary steps added by the Bush administration, but it preserves a role for the White House in reviewing draft versions of the scientific studies. During the Bush administration, the White House Office of Management and Budget began reviewing draft assessments both before and after the studies underwent an external peer review. OMB sometimes manipulated results or blocked the EPA from completing its work.

EPA’s new process refers only to the White House, not OMB specifically. An article in the most recent issue of the Watcher, OMB Watch’s e-newsletter, identifies some of the questions left unanswered in regard to the White House’s role in the new IRIS process:

EPA did not indicate why it believes White House review is necessary. It is unclear what value, if any, a White House review adds to the process, particularly after the assessment has been peer reviewed. It is also unclear who will lead the review for the White House. OMB may continue to review drafts, or other offices, such as the Office of Science and Technology Policy, may play a bigger role.

In the past, employees inside the IRIS program have expressed concern with OMB's involvement. Comments on the 2008 GAO report complained that the OMB review delayed the completion of assessments and said OMB's comments "can be very extensive and troubling to address." 


EPA does appear to want to soften the White House’s potential impact by setting time limits on the White House review and requiring public disclosure of written communications between EPA and other agencies, including White House offices.

But those measures still don’t answer the bigger question: why should the White House be involved at all? The IRIS program assesses the existing state of science regarding the potential health effects of chemicals that people are exposed to. EPA then subjects the assessments to an external peer review. The IRIS program engages in risk assessment, not risk management. In other words, this is not policy. The White House should let IRIS staff do its work in peace and allow it to publish public health information on its own terms.

(Matthew Madia 06/03/09; 0 comments)

EPA Regains Control over Chemical Assessments

 

The Environmental Protection Agency (EPA) announced today that it would take back the reins of its program for assessing the risks of industrial chemicals after the process had become corrupted by political forces.

EPA is streamlining the process for evaluating and listing chemicals under its Integrated Risk Information System (IRIS). A memo from EPA Administrator Lisa Jackson says, “While still robust, the assessment development process will be shortened to 23 months, speeding the availability of IRIS assessments to the risk assessor community and the public and providing for more timely action to protect public health.”

To shorten the process, EPA is limiting the role of other federal agencies, including White House offices. EPA made major changes to the IRIS process in April 2008 when it granted other agencies several opportunities to review draft assessments and make changes or submit new data. Largely political offices in the White House, as well as the Department of Defense, a major user of hazardous chemicals, could delay or interfere with IRIS assessments on non-scientific grounds.

The new process announced today maintains a role for the White House and other agencies but allows EPA to keep the ball in its own court. EPA manages the review process by setting deadlines for comments (45 days) and maintaining editorial control. EPA also says that comments on draft assessments should focus solely on science.

If other agencies are able to interfere in an assessment, the public will find out. EPA is requiring that all written comments on draft assessments become part of the public record.

While OMB Watch had called for a prohibition on White House review of purely scientific material, EPA’s new process seems like a reasonable compromise. Still, we’ll all need to keep a vigilant eye on the progress of the IRIS program from this point forward.

Most importantly, EPA plans to trim the length of the typical assessment to about 23 months (a lightning-quick pace, since the old agency review process in and of itself could take years). A faster assessment process will provide the public and decision makers with timely information on health hazards, allowing consumers to better avoid dangerous products and prompting regulators to intervene. In recent years, EPA had only been completing about two assessments per year. Jennifer Sass, a Senior Scientist at the Natural Resources Defense Council, says it best:

I hope that this new process will lead to more IRIS assessments being finalized each year, and more assessments of the most hazardous and most widespread chemical contaminants. These are the ones that we really need to target with solid science and health-protective policies, if we are to improve health where people live, work, and play!

 

(Matthew Madia 05/21/09; 0 comments)