Fine-Tuning LLMs for Finance: FinBERT to BloombergGPT

TL;DR

• Financial nuances require specific training: General LLMs fail on financial text due to domain-specific jargon and the need for numerical reasoning, with accuracy dropping by up to 40% on complex financial tasks.
• Purpose-built models outshine general ones: BloombergGPT achieves state-of-the-art results on financial NLP benchmarks by incorporating a 345 billion token proprietary financial dataset.
• Open-source is catching up: Quant funds are increasingly fine-tuning open-source models like Llama 3 and Mistral using targeted financial datasets to rival proprietary model performance at a fraction of the cost.

The Challenge of General LLMs in Finance

General-purpose Large Language Models (LLMs) like GPT-4 have revolutionized many industries, but they often struggle when applied directly to complex financial tasks. The reason lies in their training data, which is heavily skewed towards general internet text. Financial jargon, intricate numerical reasoning, and the strict regulatory context of financial documents are often underrepresented. As a result, general models can misinterpret subtle market signals or hallucinate facts when summarizing dense SEC filings.

Furthermore, the temporal nature of financial data poses a significant challenge. Financial markets are dynamic, and information becomes outdated rapidly. A model trained on data up to 2024 might struggle to contextualize events in 2026 accurately without ongoing fine-tuning or retrieval-augmented generation (RAG) strategies. For quantitative applications where precision is paramount, these shortcomings are unacceptable.

To overcome these limitations, the industry has turned to domain-specific fine-tuning. By continuing the training process on a curated corpus of financial literature - such as earnings transcripts, analyst reports, and news - researchers can adapt these models to understand the nuances of the financial domain. This approach is essential for applications ranging from automated sentiment analysis to complex portfolio optimization strategies.

FinBERT: Pioneering Financial NLP

One of the earliest and most successful examples of domain-specific adaptation is FinBERT. Built upon the architecture of Google's BERT, FinBERT was fine-tuned specifically on a massive dataset of financial communications. This included corporate reports, earnings call transcripts, and financial news articles. The primary goal was to enhance the model's ability to classify financial sentiment, a crucial task for algorithmic trading.

FinBERT's architecture remains fundamentally the same as BERT's - a transformer-based model designed for bidirectional representation learning. However, by updating its weights using domain-specific data, FinBERT learned to recognize that words like "liability" or "share" have very specific meanings in a financial context, unlike their broader usage in general English. This specialized understanding allows FinBERT to outperform general BERT models significantly on financial sentiment classification tasks.

The success of FinBERT proved that even relatively small, specialized models could offer substantial value in finance. It paved the way for more ambitious projects, demonstrating that targeted training could bridge the gap between general language understanding and the specialized needs of quantitative finance.

BloombergGPT: Scaling Financial AI

Building on the concepts pioneered by models like FinBERT, Bloomberg introduced BloombergGPT, representing a massive leap in scale and ambition. Rather than simply fine-tuning an existing model, Bloomberg built a 50-billion parameter language model from scratch. The key differentiator was its training dataset, a carefully constructed mix of 51% proprietary financial data (amounting to 345 billion tokens) and 49% general-purpose data.

This hybrid approach allowed BloombergGPT to maintain strong general language capabilities while achieving unprecedented performance on financial benchmarks. It excelled in tasks such as financial named entity recognition, sentiment analysis, and answering complex questions based on financial documents. The proprietary data, drawn from Bloomberg's vast archives, gave the model a unique edge that open-source alternatives struggled to match.

However, the development of BloombergGPT also highlighted the immense resources required to build such models. The computational cost and the need for massive, high-quality proprietary datasets place such endeavors out of reach for most organizations. This has sparked a debate in the quantitative community about the relative merits of building proprietary models versus fine-tuning open-source alternatives. Check out our internal guide on Multi-Agent AI Systems in Financial Research to learn more.

Fine-Tuning Open-Source LLMs for Quants

For most quantitative funds, building a model from scratch like Bloomberg is impractical. Instead, the focus has shifted towards fine-tuning powerful open-source models, such as Llama or Mistral, using parameter-efficient fine-tuning (PEFT) techniques like LoRA. This approach allows quants to adapt large models to their specific proprietary data and trading strategies without the astronomical costs of full-scale training.

The process typically involves gathering a highly curated dataset relevant to the fund's specific niche - for example, a dataset of historical options pricing data and corresponding news events. The open-source model is then fine-tuned on this data, adjusting only a small subset of its parameters. This results in a model that is deeply specialized for the fund's strategy while retaining the broad reasoning capabilities of the base LLmodel.

Evaluation is critical in this process. Funds rely on benchmarks like FinQA (Financial Question Answering) and FiQA (Financial Sentiment Analysis) to measure the effectiveness of their fine-tuning efforts. Moreover, they employ rigorous backtesting against historical market data to ensure that the model's insights translate into viable trading signals.

Benchmark Comparison

Disclaimer: This article is for informational purposes only and does not constitute financial advice.