October Surprise: Looming Recovery Act Data Quality Issues

At the end of October, the first round of recipient reporting for the Recovery Act will be released on Recovery.gov [1]. This reporting is a crucial step in Recovery Act oversight and transparency, but there is no guarantee that the reporting process will proceed smoothly. Come October, the diffusion of responsibility for Recovery Act data quality could result in a great deal of confusion, as a flood of bad data could stymie the administration’s efforts at Recovery Act transparency.

The data provided by the reporting process is supposed to let both the government and the public track where Recovery Act funds are going, and, importantly, how many jobs have been created or saved due to Recovery Act programs. Despite this vision, Earl Devaney, chair of the Recovery Accountability and Transparency Board, has been saying publicly [2] that the first round of reporting is likely to be very rough, complete with a great deal of bad data. Unfortunately, no agency has taken sole responsibility for data quality, creating a last-minute scramble as the reporting deadline approaches.

The Recovery Act reporting requirements are not exceptionally complex, and a majority of recipients should be able to report on time without incident. However, when one is dealing with tens of thousands, potentially hundreds of thousands, of recipients, even a 10 percent error rate would result in an enormous amount of erroneous information, creating an equally large PR headache for the administration.

The problems with reporting will likely attract media attention. Already, critical articles on the poor quality of Recovery Act data have been written, including a recent New York Times article [3] highlighting the "fuzzy math" behind some construction projects in New York City. Also, some articles will focus on minor errors that appear as wasteful spending, like the recent "$1 million sliced ham" [4] stories, which reported on a poorly described Recovery Act procurement that made it seem like the government spent $1.19 million on two pounds of sliced ham. (The description of the procurement failed to mention that the order was for 760,000 sliced hams, not just one ham.)

The reaction to Recovery Act spending will be the result of several issues that will arise in October. First, there is the possibility that some recipients might not report, limiting the information available about Recovery Act spending. Theoretically, agencies will know the prime recipients they have given stimulus funding to, but they will not know any subrecipients those primes divide their funding among until data is reported. Therefore, it will be very difficult for agencies to know whether all the subrecipients that are required to report have done so. This is because the prime recipients are not required to disclose if and when they will divide their funding among subrecipients until the information is reported at the end of each quarter. Agencies will then be left with a short 10-day period to ensure all recipients have reported data.

Another cause for concern will be the quality of the data from those recipients who do complete their reports. It will be the first time Recovery Act recipients are using the reporting system, and there are bound to be problems when they report. The data dictionary [5] the Office of Management and Budget (OMB) released is a technical document, which could be confusing for organizations that are not used to federal reporting guidelines.

There are also several data points which require subjective, often narrative entries, which will result in a wide, uneven range of answers. Several of these entries involve job creation and retention estimates, which will be the most scrutinized entries in the entire data set and yet are the most likely to have errors. For example, one report field requires recipients to estimate the number of jobs created, despite the administration giving relatively little guidance on how to differentiate between jobs created versus jobs saved, as well as the lack of a definition of what constitutes a job. The administration has left it up to the individual agencies to disseminate statistical techniques for their recipients to use, and less than half of the federal agencies have done so thus far. The lack of central guidance in job creation estimates that will come from this reporting cycle will provide critics with more ammunition to criticize the Recovery Act.

How have these problems arisen? If the data, which are the foundation of oversight, are of poor quality, then one would be tempted to blame the Recovery Board, since it is specifically charged with Recovery Act oversight. However, Devaney has repeatedly stated [6] the board's mission is data integrity or protection, not data quality. Devaney has said it would be inappropriate for the board, as a collection of Inspectors General, to become involved in the actual collection of data. Instead, Devaney points to OMB and the agencies, arguing that they change and execute the reporting guidelines.

OMB, however, points to the agencies and Recovery Act fund recipients. According to OMB’s guidance documents [7], the agencies must take responsibility and work with their recipients to ensure comprehensive and accurate data reporting.

Recently, however, OMB has begun to take greater responsibility for data quality. At a Senate hearing on Sept. 10, OMB Deputy Director Rob Nabors said OMB is working to make sure the reporting process goes as smoothly as possible. Nabors detailed steps the agency is taking – such as sending OMB personnel to state and local municipalities to facilitate communication between recipients, agencies, and OMB – and holding webinars for Recovery Act recipients.

More importantly, on Sept. 11, OMB released a new memorandum to the federal agencies titled "Improving Recovery Act Recipient Reporting [8]." The memo "identifies essential actions that Federal agencies must take immediately to effectively assist recipients in meeting reporting requirements" and may signal that OMB is concerned the first round of recipient reporting will be difficult. OMB's memo says "current recipient registration is below expected levels, which may lead to underreporting" as the reporting deadline approaches, and that the agencies must start identifying their recipients now. By identifying the recipients well ahead of time, OMB is trying to avoid the missing-recipients problem.

However, it remains to be seen how effective these steps will be and if they are simply too late in the process to be effective. With less than a month remaining until the reporting deadline, agencies still might not have enough time to perform the outreach OMB is recommending.

It appears that both OMB and the Recovery Board understand the October reporting period will be rough. The board had originally planned on releasing the data on Oct. 11, the day after the reporting deadline, but it recently decided [9] to hold onto the information a little longer, consistent with the intent of the Recovery Act, which builds in a 20-day error correction period. According to the Recovery Board, the recipient reports collected by Oct. 10 will be released on Oct. 30. Additionally, information about contracts provided directly from federal agencies will be posted to Recovery.gov on Oct. 15. The agencies will be able to use these additional days to correct simple errors in the data before it is released, ensuring a more accurate representation of Recovery Act spending in a timely manner.