BFI NewsNov 17, 2016

Mining Text to Measure Uncertainty

Steve Davis explains the methodology behind a valuable index

Shocks to the economy, such as the 2008-09 financial meltdown and the subsequent sovereign debt crisis, ratchet up uncertainty and anxiety about the economic future. This economic distress can lead to populist pressure for political change.

Likewise, the political process itself can generate uncertainty, as in the case of protracted US budget policy fights over the debt ceiling in 2011 and the fiscal cliff episode in 2012. Globally, the Brexit vote in the United Kingdom and financial and political events in China are big unknowns.

Whatever the source, this policy uncertainty takes an economic toll, Steven J. Davis told undergraduates in a Nov. 11 Friedman Forum talk. “There is a large literature that suggests elevated levels of uncertainty are likely to discourage investment and consumption and increase asset volatility,” noted Davis, the William H. Abbott Professor of International Business and Economics at Chicago Booth.

How big is that effect? Quoting Lord William Thompson Kelvin from 1883, Davis said the first essential step to finding out was to be able to measure the level of uncertainty itself. Davis showed how, with Scott R. Baker of Northwestern University and Nicholas Bloom of Stanford University, he used textual analysis of newspaper stories to develop a measure known as the Economic Policy Uncertainty index.

They analyzed 10 major US newspapers to tally a monthly count of articles that contain at least one term from each of three sets of keywords: terms referencing the economy, terms related to policy, and words about uncertainty. They scaled the count of articles for each paper and month as a share of all articles published in the same paper and month, and normalized the index to a level of 100 in 2009.

Undergraduates played a key role in the work, David told the crowd. The keyword tally relied on automated text analysis, but student research assistants provided human validation. Teams of RAs were trained and tested, then assigned to read and code 12,000 randomly selected newspaper articles. That step confirmed the word combinations found in the articles judged to be most relevant so researchers could hone in on searching for the most common terms.

The EPU index incorporates several other non-media measures, “but today I wanted to concentrate on computer automated newspaper search, because I want you to see just how powerful these techniques are to address other economic issues,” he said.

To confirm that the method produced a meaningful measure, the researchers replicated the text analysis with stock market terms and compared the results to the VIX volatility index; the two tracked very closely, he said.

The result is a timeline of uncertainty that now goes back to 1900 for the US and the UK. It shows peaks of uncertainty around elections, the Sept. 11 attacks and the Gulf Wars. The collapse of Lehman Brothers and the implementation of the Troubled Asset Relief Program scored higher than the Gulf War. The US index hit its highest levels in 2011 and 2012 during the debate over the debt ceiling and fiscal cliff.

The team has extended the work to create topic-specific measures of uncertainty related to health care policy and national security. In new work, Davis has created a monthly global uncertainty index incorporating 16 nations that account for two-thirds of global economic output. Tracked back to 1997, the global index shows that uncertainty has been at a sustained high for the last five years—at levels 60 percent higher than in the previous fourteen and one-half years and 22 percent higher than in 2008–09.

The indexes have been widely used by other researchers to correlate actual economic performance with the movement of public uncertainty.

Speaking just days after the 2016 presidential election, Davis said that the campaign and Trump’s surprising victory had no doubt led to rising uncertainty, especially around the issues of trade, immigration, taxes, and regulation. And the post-election spike was dramatic. “Even though we’re using a seven-day rolling average, in three days of postelection outcome, the index went from around 100 to 323 and 355,” Davis said. “If stays at that level for a month, that will be higher than anything we had.”

The index has proved to be a useful tool for researchers working to quantify how businesses, investors, and consumers respond to uncertainty; it has been used to test the notion that uncertainty played a role in the slow recovery and sluggish growth since the financial crisis.

Other researchers have used the text analysis methods to measure changes in partisan language and political polarization. More recently, Chicago Booth’s Tarek Hassan used the method to analyze transcripts of corporate earnings teleconferences to measure uncertainty within firms and track their responses.

Financial firms have also adopted the methodology to estimate uncertainty, Davis added. “There seems to be high demand for our product –but we give it away, so it’s popular but at a price of zero,” he joked.

Toni Shears