(John Kemp is a Reuters market analyst. The views expressed are his own)
By John Kemp
LONDON (Reuters) - A thoughtful new paper from researchers at the University of Illinois marks a significant step forward in research on how commodity futures prices are formed.
Until recently, the academic and policy debate about futures price formation has been locked in an acrimonious and polarized standoff between market fundamentalists, who insist all price moves reflect supply and demand fundamentals, and those writers who blame speculators for every rise in food and fuel prices.
Both views tend to be colored by the policy outcomes researchers favor. Anti-poverty campaigners focus on the role of speculation because they want governments to impose more controls on the cost of food and fuel. Free-market economists stress the role of fundamentals to deny governments any ammunition to meddle.
Both positions are extreme and unconvincing.
Now Xiaoli Etienne, Scott Irwin and Philip Garcia have published an innovative paper examining the evidence for temporary price bubbles in markets where prices are otherwise driven by fundamental factors.
According to the authors, futures prices for grains, livestock and soft commodities like sugar have all exhibited multiple bubbles over the last four decades, with bubbles more common in the 1970s and again in the 2000s than during the 1980s and 1990s.
Bubbles pre-date the rising popularity of indexing strategies and the "financialisation" of commodity markets. There is no evidence bubbles have become more frequent or larger following the entry of more financial investors into commodity futures markets since 2005.
"Bubbles existed long before commodity index traders arrived and the process of commodity market financialisation started," according to a paper on "Bubbles in Food Commodity Markets: Four Decades of Evidence" presented at an IMF seminar in Washington on March 21.
In fact most of the biggest and long-lasting bubbles occurred in 1971-76. Financialisation may have ensured bubble-like price movements are now smaller and reverse more quickly.
"Compared to the post-2000 years, speculators and irrational traders (may have) played a greater role influencing prices in the 1970s because markets were less actively traded. The arrival of new traders in recent years, coupled with a dramatic increase in trading volumes, has increased market liquidity, apparently reducing the frequency of bubbles," the authors write (http://www.imf.org/external/np/seminars/eng/2012/commodity/pdf/irwin.pdf).
The persistence of bubbles remains perplexing. The authors speculate bubbles may be driven by herding behavior, momentum trading or other "noise traders".
"One possible explanation may be that markets are sometimes driven by herd behavior unrelated to economic realities ... As markets overreact to new information, commodity prices may thus show excess volatility and become explosive."
"It may also be that there are many positive feedback traders in the market who buy more when the price shows an upward trend and sell in the opposite situation. When there are too many feedback traders for the markets to absorb, speculative bubbles can occur in which expectations of higher future prices support high current prices."
"It may be fads, herding behavior, feedback trading, or other noise traders that have long plagued futures markets were highly influential in recent price behavior. Recent empirical evidence does suggest that herding behavior exists in futures markets among hedge funds and floor participants."
The paper concludes with an appeal for more research to identify the source of bubble-like price behavior.
GREAT LEAP FORWARD
In most other asset classes, it is now accepted market prices are basically driven by fundamentals, especially in the medium and long run, but in the short term can department from them, sometimes significantly, as a result of speculative factors.
As billionaire investor Warren Buffett noted in 1988 about the hardline believers in efficient market theory: "Observing correctly that the market was frequently efficient, they went on to conclude incorrectly that it was always efficient. The difference between these propositions is night and day."
Now a new generation of researchers are developing theories which allow for a combination of both fundamental and speculative factors to affect commodity futures prices.
Etienne, Irwin and Garcia's paper is a big step forward because it carefully distinguishes between the influence of the commodity index traders and the short-term bubbles evident in commodity futures prices.
It also shows how behavioral factors could be integrated into a fundamental theory of commodity futures pricing, as has been accepted in every other major asset class.
Crucially, it shows how herding, momentum-based trading strategies and other noise trading may cause futures prices temporarily to depart from fundamentally determined levels, but suggests such deviations have been relatively brief and reversed within weeks or months.
Bubbles may always have been part of the operation of futures markets. For a sample of around 40 annual contracts in 12 different commodities, the authors found bubble-like behavior in about a third of the contract months studied. Bubbles occurred most frequently in sugar (55 percent of contracts studied) and least frequently in feeder cattle (25 percent) and wheat (24 percent).
BEHAVIOUR AND FUNDAMENTALS
The University of Illinois' Department of Agricultural and Consumer Economics is one of the most respected institutions in the field of commodities and derivatives, so the findings cannot be readily dismissed.
In some ways, commodity research is catching up with developments elsewhere. The formation and subsequent collapse of bubbles in other markets has been extensively studied by George Soros ("The Alchemy of Finance" 1987), Didier Sornette ("Why Stock Markets Crash" 2003) and Robert Shiller ("Irrational Exuberance" 2009) for 25 years.
Etienne, Irwin and Garcia have shown how the same approach could help improve understanding of commodity futures markets.
The authors observe "speculative bubbles are not isolated phenomena in agricultural markets, but appear in other futures markets including energy and metals markets as well" citing work by Phillips and Yu ("Dating the timeline of financial bubbles" 2011) and Gilbert ("Speculative Influences on Commodity Futures Prices" 2010).
The paper does not investigate energy futures markets. But the approach could be usefully applied to see if energy and metals markets exhibit similar bubble phenomena.
A theory of commodity price formation that embraces both fundamental and behavioral factors would provide a much richer and more realistic understanding of how futures prices are set.
Accepting that bubbles occur in food (and possibly fuel) prices does not mean they should be regulated out of existence.
Bubbles may be an integral part of the normal process of price formation in any financial market, including commodities, as investors grope towards an equilibrium in the face of incomplete information and limited liquidity.
Trying to eliminate bubbles through regulation may do more harm than good. Accepting temporary bubbles may be the price for allowing the market to perform its long-term function of price discovery.
But the paper should finally move the academic and policy debate beyond its polarized focus on whether speculation impacts commodity futures prices to ask a more nuanced question: how?
(Editing by William Hardy)