If you want a picture of the future, imagine asking a large language model for a prediction.
Two sets of researchers did so recently and found that large language models (LLMs) like ChatGPT and BERT can enhance the accuracy of predictions about the stock market and public opinion, at least as measured against historical data.
In a paper titled, “Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models,” University of Florida professors Alejandro Lopez-Lira and Yuehua Tang evaluated how OpenAI’s ChatGPT fared when assessing the sentiment of news headlines.
Sentiment analysis – determining whether text like a news headline expresses positive, neutral, or negative sentiment about a subject or company – has become a widely evaluated parameter for quantitative analysis algorithms used by stock traders. It has been found to make market predictions more accurate.
The two University of Florida boffins looked at how ChatGPT performed when prompted to assess the sentiment expressed in news headlines. When they compared ChatGPT’s evaluation of those news stories to the subsequent performance of company shares in their sample, they found the model returned predictions that were statistically significant, which is more than can be said of other LLMs.
“Our analysis reveals that ChatGPT sentiment scores exhibit a statistically significant predictive power on daily stock market returns,” they state in their paper.
“By utilizing news headline data and the generated sentiment scores, we find a strong correlation between the ChatGPT evaluation and the subsequent daily returns of the stocks in our sample. This result highlights the potential of ChatGPT as a valuable tool for predicting stock market movements based on sentiment analysis.”
For example, they prompted ChatGPT thus:
In the paper, ChatGPT responded:
The researchers interpret this to mean that ChatGPT’s analysis assumes the fine could nudge up Oracle’s sales and stock price.
As detailed in the paper, ChatGPT did a better job analyzing sentiment than other LLMs, specifically GPT-1, GPT-2, and BERT.
“The superiority of ChatGPT in predicting stock market returns can be attributed to its advanced language understanding capabilities, which allow it to capture the nuances and subtleties within news headlines,” the researchers conclude.
“This enables the model to generate more reliable sentiment scores, leading to better predictions of daily stock market returns.”
Your mileage, however, may vary, which is to say that you’ll want to know how your model’s temperature parameter, which affects response randomization, is set. When The Register entered the same prompt using the free web interface to ChatGPT today, we received the opposite answer:
And then a second time, with an extra carriage return between the prompt and headline, the answer was:
Asked about this, Alejandro Lopez-Lira, assistant professor of finance at University of Florida and one of the paper’s co-authors, speculated the web interface is more random than the paid-for API.
Lopez-Lira told The Register in an email that ChatGPT alone is not sufficient for sentiment analysis on current events.
We think ChatGPT has lots of room for improvement
“We think ChatGPT has lots of room for improvement,” Lopez-Lira said in an email. “ChatGPT, for example, does not have the latest information on COVID or the war. That’s why we think of this as a baseline of what models can do. Providing more context either in the prompt or by fine-tuning will probably make the models better at forecasting. In some sense, what we show is a lower bound on the capabilities.”
Sentiment analysis by itself is not a strong indicator of stock price movement, though it still has value for stock traders.
“The correlation is very small but statistically significant,” said Lopez-Lira.
“It is on the order of one percent. However, because these are daily correlations for multiple stocks, they quickly result in high returns. For example, (without transaction costs) it results in a Sharpe ratio at least twice the market’s.”
He added, “Most of the movements in the stock market are not related to direct news about fundamentals but rather change investors’ risk tolerance (sentiment) or their future expectations. We think adding contextual information on the sentiment of the market will probably make return predictability stronger.”
In a separate paper, “Language Models Trained on Media Diets Can Predict Public Opinion,” MIT researchers Eric Chu, Jacob Andreas, and Deb Roy, along with Harvard researcher Stephen Ansolabehere, found large language models trained on specific media (online news, TV broadcasts, or radio) can predict the opinions of groups exposed to that media.
“With ‘media diet models,'” explained co-author Eric Chu, a Google research scientist who was an MIT doctoral candidate at the time of the research project, via Twitter, “we predict how a group of media consumers will answer polls by training a [language model] on media they consume.”
These media diet models were based on BERT, a widely known large large language model, and fine-tuned with a media diet dataset.
The authors said their work points the way to more accurate public opinion polling, but also invites further examination of how media affects people and shapes public opinion.
They argue for media diet-specific analyses that examine: “(1) selective exposure, or the general systemic bias in which people gravitate towards information that is congruent with their prior beliefs; (2) echo chambers, in which the selected environments amplify and strengthen opinions shared with like-minded individual; and (3) filter bubbles, in which content curation and recommendation algorithms surface items based on users’ past behaviors, again confirming the users’ worldviews.”
“Media diet models could help identify subpopulations being exposed to potentially harmful messaging,” the authors suggest.
That’s the best case scenario. They may also prove useful to media manipulators for assessing the effectiveness of their disinformation campaigns. ®
