A recent academic paper published in the Journal of Economic Perspectives (Volume 29, Number 4, Fall 2015), “Household Surveys in Crisis,” illustrates the problems of government surveys that are used to make multibillion dollar and multiyear decisions of government and business. We had previously highlighted the problem in a 2013 article published in the Wall Street Journal “Monthly Economic Data Aren't Reliable” which stated “economic policy and business decisions are made with initial data that are plainly inaccurate—and this can lead to mistakes, including hasty changes in direction.”

The chart shows a major part of the problem: increasing nonresponse rates. Government data collection typically gets very high cooperation rates compared to other surveys. As seen in the chart, some of the nonresponse rates have doubled. This is not always a problem in terms of the final results, but it increases the doubt and skepticism that data accurately reflect actual conditions. We have always advised taking longer term views of data, not acting on the last data released, and paying careful attention to data revisions. As managers, we often need to act quickly in response to events. Concerns about data increase the value of personal experience of managers. Just because data appear to have surface validity, it's essential to ask hard questions, and seek cross-validation from multiple sources.