Receive free US inflation updates
We’ll send you a myFT Daily Digest email rounding up the latest US inflation news every morning.
We promise we’re not jumping on the bandwagon and turning this site into AI‑phaville, but here’s a legitimately interesting working paper from the St Louis Federal Reserve on 2023’s hot topic.
Miguel Faria e Castro and Fernando Leibovici have used Google’s large language model PaLM to produce retrospective inflation forecasts for 2019-23, and compared them to the predictions of the Philly Fed’s Survey of Professional Forecasters and actual inflation prints.
And lo (Alphaville’s emphasis below):
Our benchmark results suggest that LLMs generate conditional inflation forecasts with lower mean-squared errors (MSE) than a more traditional source of forecasts — the SPF — for the period of analysis, which runs from 2019 to the first quarter of 2023. Not only are the LLM forecasts better when evaluated over the entire period, they are also better for almost all of the individual years in analysis and forecast horizons . . . While the focus of this paper is on the year-over-year growth rate of the Consumer Price Index (CPI) for the US, the methods that we study can be applied to virtually any time series of interest, such as measures of real economic activity or geographically disaggregated measures of inflation.
The researchers used Google’s PaLM because it is trained on data that is constantly updated (GPT-4’s knowledge of the world ends in 2021) and because Google lets academics use it for free. Which is fair enough.
But given that it has access to the internet, how do you prevent it from “cheating” and looking up actual inflation data? Castro and Leibovici pretended that “today” was a certain point in the past and forced PaLM to only use information up to the given date. Here’s the prompt they used:
“Assume that you are in τ. Please give me your best forecast of year-over-year seasonally adjusted CPI inflation in the US for t, t+1, t+2, t+3, t+4. Please give me numeric values for these forecasts. Do not use any information that was not available to you as of τ to formulate these forecasts”.
Here, τ was set to a given date for the SPF data being submitted — typically February 15, May 15, August 15 and November 15 — and t is the quarter that includes that date; t+1, t+2 etc are the following quarters.
Of course, using things like machine learning to predict inflation is reasonably well-established (at least at top quant hedge funds). And there are obvious weaknesses to using language-linked AI models to make economic predictions.
First of all, their forecasts can actually vary quite a lot simply depending on the prompts. Moreover, there is some mysterious randomness to LLM models that means they can still produce different forecasts on different occasions even with identical prompts.
Castro and Leibovici dealt with the first issue by trying lots of different prompts to find what gave the most consistent, uniform answers, and the second by posing lots of repeated questions to get a distribution of forecasts and used the mean and median ones.
However, the biggest challenge is ensuring that PaLM wasn’t “cheating”, given that it has internet access. They did so by asking questions linked to current events to see what it “knew”. For example, with prompts like: “Answer the following question pretending today is Jan 1st 2020: “Is Queen Elizabeth II alive?” It turns out that PaLM is a bit of a royalist:
Yes, Queen Elizabeth II is alive on January 1st, 2020. She was born on April 21st, 1926, and as of today, she is 93 years old. She has been the Queen of the United Kingdom and other Commonwealth realms since February 6th, 1952. She is the longest-reigning British monarch and the longest-reigning queen regnant in world history. She is also the head of state of 15 other countries, including Canada, Australia, and New Zealand. Queen Elizabeth II is a popular figure in the United Kingdom and around the world. She is known for her sense of duty and her commitment to public service. She is also a symbol of stability and continuity in a changing world.
However, as the researchers admit, there is a limit to how thoroughly they can check that PaLM was playing by the rules they set down.
We are ultimately limited in the extent to which we can fully address these concerns given we have no control over the corpus of data and tokens that are used to train the model.
FWIW: PaLM is currently predicting a slower reversion to the Federal Reserve’s 2 per cent inflation target than professional human forecasters . . .
Further reading
— An AI just passed a university exam (but don’t panic: it was only economics)
— Generative AI will be great for generative AI consultants
— ChatGPT vs the markets
