Financial sentiment analysis: Classic methods vs. deep learning models

3.4.1BERT

BERT (Bidirectional Encoder Representations from Transformers) [82] is a state-of-the-art language representation model that leverages the Transformer architecture [83]. As a pre-trained model, BERT is able to capture the contextual relationships and meanings of words in sentences and can also be fine-tuned for specific NLP tasks such as text classification, question-answering, and more. Moreover, the term bidirectional refers to the model’s ability to consider both the left and right context of a word when determining its meaning, leading to better contextual embeddings. There are two primary iterations of the BERT model: the Base model, characterized by 12 layers, 768 hidden states, 12 attention mechanisms, and a total of 110 million trainable parameters; and the Large model, comprising 24 layers, 1024 hidden states, 16 attention mechanisms, and a total of 340 million trainable parameters. The BERT Large model possesses enhanced capabilities for capturing intricate linguistic relationships, albeit at the cost of heightened computational resource demands.

BERT undergoes a two-stage training process. In the first stage, it undergoes pre-training through a dual-task framework encompassing Masked Language Modeling and Next Sentence Prediction. This initial phase capitalizes on the vast textual resources of the BooksCorpus (800 million words) and English Wikipedia (2.5 billion words). The primary objective of this pre-training stage is to equip the model with a robust understanding of general language patterns and structures. Subsequently, in the second stage, BERT is fine-tuned for specific NLP tasks, such as SA. During fine-tuning, BERT is exposed to task-specific datasets containing labelled data. In this stage, task-specific layers are added on top of the pre-trained BERT model. These layers may include feedforward neural networks and output layers tailored to the specific task. Moreover, during fine-tuning, the pre-trained BERT weights are updated to adapt to the task at hand. This process enables the model to adjust its pre-trained, generalized language representations to the intricacies of the target task.

Notably, BERT has demonstrated exceptional performance across a spectrum of NLP benchmarks, including the General Language Understanding Evaluation (GLUE) benchmark, Stanford Question Answering Dataset (SQuAD v1.1), SQuAD v2.0, and the SWAG dataset, reaffirming its prowess in language understanding and comprehension [82]. Note that although this model has not been designed for economic texts, it has nevertheless been used among other models in FSA research papers with quite good results [84, 85].

3.4.2RoBERTa

RoBERTa (A Robustly Optimized BERT Pretraining Approach) [86], represents an advanced language representation model that evolved from the BERT architecture. This evolution involved a series of meticulous adjustments, including extended training duration, exposure to an augmented training dataset, heightened batch size, and the incorporation of longer sequences, alongside the deliberate omission of the Next Sentence Prediction (NSP) task in favor of dynamic masking. These strategic refinements, crucially, do not compromise the model’s classification capabilities. Quite the opposite, RoBERTa demonstrates superior performance in comparison to its predecessor, BERT, across a spectrum of key NLP benchmarks, including SQuAD v1.1, SQuAD v2.0, the General Language Understanding Evaluation (GLUE) benchmark, RACE, MNLI-m, and SST-2.

One remarkable facet of RoBERTa’s training regimen is its exposure to a substantially enlarged corpus of text data. This corpus encompasses additional datasets like CC-News, Open WebText, and STORIES, aggregating a vast collection totalling 160 GB [86]. This rich and diverse training data amplifies RoBERTa’s language comprehension prowess significantly. Like original BERT, RoBERTa has not been designed for economic text, but also have been used also in FSA research papers with very remarkable performance [87, 88, 89].

3.4.3Financial BERT models

FinBERT models are specialized language representation models designed for financial applications and built upon the BERT architecture. These models are explicitly trained to analyze and comprehend textual data pertaining to financial markets, stocks, investments, and economic news. Their primary objective is to evaluate sentiment and extract valuable insights from financial text. Similar to BERT, the typical training process for FinBERT includes an initial pre-training phase. During this phase, the model acquires general language representations by processing a substantial corpus of financial text data. This pre-training stage enables the model to capture specific language patterns and financial terminology relevant to its domain. Following pre-training, FinBERT undergoes fine-tuning, focusing on FSA tasks. Fine-tuning entails training the model on specialized datasets containing labelled data, refining its performance for sentiment analysis within the financial context. It is worth noting that within the term “FinBERT,” the literature identifies four distinct models that we are aware of. The first model was introduced by Araci [63], while the subsequent three models were developed by Desola et al. [64], Liu et al. [65], and Yang et al. [2, 67], respectively. These variations reflect the evolving landscape of research and applications within the domain of FSA.

The FinBERT model from Araci [63] was pre-trained on the financial subdataset of Reurers dataset TRC2. It was then evaluated on the Financial Phrase Bank and FiQA Sentiment datasets, i.e. the same datasets used in this paper. The only difference lies in the evaluation, where this model was evaluated with the original FiQA dataset, where sentiment values range continuously from -1 to 1. In the existing literature, this model seems to outperform all the other testing models in accuracy and F1 score, even those that are more suitable for the financial domain, such as the LPS model [46], FinSSLx [48], HSC model [90]. Implementation of this model can be found via Hugging Face AI repository [91].

The FinBERT model developed by Desola et al. [64] comprises three distinct variants: FinBERT Prime, FinBERT Pre2K, and the combination of both, referred to as the FinBERT Combo model. FinBERT Prime was pre-trained on 10-K filing reports, specifically those submitted by companies to the SEC and accessible via the EDGAR system, spanning the years 2017 to 2019. In contrast, FinBERT Pre2K underwent pretraining on 10-K filings dating back to 1998 and 1999. The FinBERT Combo model benefits from the fusion of data from both of these datasets. According to the authors, their FinBERT model demonstrates superior performance when compared to the standard BERT model, particularly in Next Sentence Prediction and Masked Language Modeling tasks. For those interested in implementing these models, the necessary resources and code can be found on Github [92].

The FinBERT model, as developed by Liu et al. [65], underwent a comprehensive pre-training process utilizing three substantial financial datasets: a) The Financial Web dataset (comprising 24GB of text and 6.38 billion words). b) Yahoo! Finance dataset (with a size of 19GB and 4.71 billion words). c) RedditFinanceQA dataset (amounting to 5GB and 1.62 billion words). Following this extensive pre-training phase, the model underwent fine-tuning and subsequent evaluation on two critical datasets: the Financial Phrase Bank and the original FiQA Sentiment datasets. Remarkably, the results of this implementation showcase a noteworthy performance improvement compared to the conventional BERT model. Furthermore, it’s noteworthy that this particular model exhibits superior performance when compared to the FinBERT model by Araci [63], demonstrating its prowess across both the PhraseBank and FiQA datasets.

The FinBERT model, as proposed by Huang et al. [2, 67], comprises four distinct variants: the initial pair includes FinBERT-BaseVocab, both in uncased and cased versions, while the subsequent pair encompasses FinBERT-FinVocab, also in uncased and cased versions. This comprehensive model family underwent pre-training on three distinct financial corpora, namely Corporate Reports (10-K & 10-Q), Earnings Call Transcripts, and Analyst Reports. A pivotal distinction arises between the BaseVo-cab and FinVocab subfamilies: the former employs the original BERT base model, pre-trained on the aforementioned financial corpora, while the latter is fashioned from the ground up, utilizing a newly crafted financial vocabulary. These resulting models underwent subsequent fine-tuning and rigorous evaluation, encompassing datasets such as the Financial Phrase Bank, AnalystTone, and the FiQA Sentiment dataset. Notably, in the case of the FiQA Sentiment dataset, a conversion from regression to classification was implemented, mirroring the methodology utilized in the present study. Remarkably, the FinBERT-FinVocab uncased model exhibits superior performance, surpassing not only its predecessor BERT but also outperforming deep learning models like LSTM and CNN. Additionally, it outshines classical ML models such as Naive Bayes, Support Vector Machines, and Random Forests across multiple metrics, including accuracy, precision, recall, and F1 score. For those interested in implementing these models, the requisite resources and code can be accessed through the Hugging Face repository [93].

Another notable BERT-based model tailored for the financial domain is FinancialBERT, as introduced by Hazourli [94]. It is worth noting that, to our knowledge, this model has not been officially presented at any conferences or published in a journal; instead, it is available through the academic social network known as ResearchGate. FinancialBERT underwent pre-training on a comprehensive array of four financial datasets, encompassing the TRC2 financial sub-dataset, Bloomberg Financial News spanning from 2006 to 2013, Corporate Reports sourced from the EDGAR database, and Earnings Call Transcripts. According to its creator, this model exhibits remarkable performance, surpassing not only the BERT base model but also outperforming the FinBERT model by Huang et al. [2, 67] on the Financial PhraseBank dataset. For instance, the aforementioned FinBERT model achieves an accuracy of 0.87 and an F1 score of 0.85, whereas FinancialBERT excels with scores of 0.99 and 0.98, respectively. For those interested in implementing this model, resources and code can be accessed through the Hugging Face AI repository [95].

In the Hugging Face AI repository, several implementations of FinBERT models are available. However, providing a comprehensive presentation or conducting exhaustive testing of these models is beyond the scope of this current paper. To facilitate a comparative analysis of various general and domain-specific Deep Learning Pre-Trained models, we will conduct the following experiments:

FinBERT (Desola et al. model) [64] and FinBERT (Liu et al. model) [65] are not part of the aforementioned series of experiments. The reason for their exclusion is that these models appear to be unavailable through the Hugging Face AI repository. Note here that to deploy all the aforementioned Deep-Learning Pre-Trained models, the Transformers library, developed by Hugging Face was used to drawn the corresponding models and tokenizers. Moreover, in the training phase of the models, ADAM optimizer was used [96].

3.5.1Accuracy

Accuracy is defined as the overall correctness of the model and mathematically as the ratio of correctly classified predictions to the overall number of predictions. The mathematical formula for computing Accuracy is:

While accuracy is straightforward to understand or interpret and clearly indicates how well a model performs in correctly classifying instances, it can be misleading when dealing with imbalanced datasets, where one class significantly outnumbers the others. Accuracy alone does not provide insights into why a model makes specific errors or which classes it struggles with.

3.5.2Recall

Recall or Sensitivity is defined as the ratio of positive predictions to the actual positive ones. The mathematical formula for computing Recall is:

Due to its inherent characteristics, this metric primarily emphasizes accurately identifying positive instances. Consequently, it proves particularly valuable when the cost of missing a positive instance (false negative) outweighs the cost of mistakenly classifying a negative instance as positive (false positive). Furthermore, recall underscores the model’s proficiency in capturing minority classes and is generally less susceptible to the effects of imbalanced datasets compared to accuracy. However, since this metric primarily centres on true positives, it offers limited insight into true negatives. Therefore, it is advisable to use it in conjunction with other metrics, such as precision or the F1-score, to achieve a more comprehensive evaluation.

3.5.3Precision

Precision indicates the ratio of predictions classified as positive and being actually positive. The mathematical formula for computing precision is:

As precision focuses on the accuracy of positive predictions, it is particularly useful when false positives are costly, such as in tasks where making mistakes has significant consequences. Combining with recall can give a better insight into the predictions, as high precision and low recall indicate that the model is cautious about making positive predictions and avoids false alarms. Using precision in isolation can lead to an incomplete evaluation of the model. One of the critical drawbacks is that optimizing for high precision may lead to missed positive instances (false negatives), especially in situations where it’s vital to identify all positive cases.

3.5.4F1-score

F1-score is defined as the harmonic mean of Recall and Precision. It is considered to be an important metric for achieving the balance between precision and recall. The mathematical formula for computing the F1 score is:

F1 score can be a more informative metric than accuracy in cases where the dataset is imbalanced, as it provides the model’s ability to classify minority class instances correctly. It is particularly useful when you want a comprehensive view of a model’s classification accuracy and condenses the evaluation of its performance into a single value, simplifying model comparison. On the other hand, like precision and recall, the F1 score primarily focuses on positive predictions and does not provide information about the model’s ability to classify negative instances correctly. It should be used in conjunction with other metrics for a complete evaluation.

3.5.5Matthews correlation coefficient

Matthews Correlation Coefficient, or MCC, is a classification performance metric that takes values from -1 (worst performance) to 1 (best performance). It is given by the following mathematical formula:

It is considered to be a more efficient and balanced metric for imbalanced classes than the F1 score and accuracy [79]. In fact, MCC considers all four outcomes of a confusion matrix (true positives, true negatives, false positives, and false negatives) in its calculation. This makes its formula more complex than some other metrics, but in the era of computational machines, this problem can be easily handled. On the other hand, the use of all kinds of outcomes makes it robust and less sensitive to imbalanced datasets.

3.5.6Receiver operating characteristic area under the curve (ROC-AUC)

ROC-AUC measures the area under the Receiver Operating Characteristic curve, which is a graphical representation of a model’s performance across different classification thresholds. It quantifies the model’s ability to distinguish between positive and negative classes, regardless of the threshold chosen. ROC-AUC is not typically expressed by a simple mathematical formula like some other metrics. Instead, it is calculated by plotting the Receiver Operating Characteristic (ROC) curve and then calculating the area under that curve. ROC-AUC is a valuable metric for assessing a model’s ability to discriminate between classes across different thresholds, especially when threshold selection is flexible or when dealing with imbalanced datasets.

4.1.1Classical ML classifiers + BoW + SMOTE

In the initial experimental setup, CatBoost outperforms all other methods, both individual and in ensemble configurations. The sole exception arises in the case of the AUC metric, where the ensemble CatBoost+LR+XGBoost achieves the highest score, albeit with a marginal variance. LR and XGBoost follow Catboost, with slightly inferion performance, reaching second and third place among individual algorithms. Regarding ensembles, the combination of CatBoost+LR+XGBoost outperforms other composite methods by attaining the highest scores across all metrics except for precision and F1. This is closely followed by the CatBoost+XGBoost+SVM ensemble, which secures the second-best performance within the ensemble category. Subsequent ensembles follow suit, generally exhibiting superior performance compared to individual methods. The full results considering this scenario are presented in Table 2.

4.1.2Classical ML classifiers + TF-IDF + SMOTE

Concerning the second scenario in which classic ML methods are used along with TF-IDF and SMOTE techniques, the combination of CatBoost+LR+SVM comes first in terms of AUC, Precision and F1 score and second in all other metrics, while the single Catboost method scores slightly higher in accuracy and recall. It should be noted here that in imbalanced datasets, like the one in our case, the examination of recall, precision and F1 gives better insight into the results than accuracy, as these metrics are particularly useful when dealing with imbalanced classes. Precision measures how many of the predicted positive instances are actually positive, and recall measures how many of the actual positive instances were correctly predicted. F1-Score is the harmonic mean of precision and recall and helps find a balance between the two. In addition, among the standalone methods, CatBoost and LR demonstrate superior performance relative to others. As for ensembles, a blend of CatBoost and LR with SVM and Ridge, respectively, emerges as the configuration that yields the most favorable outcomes. In conclusion, it is evident that ensembles consistently yield improved performance compared to individual methods, leading to slight enhancements across all metrics. The corresponding results, concerning all metrics, are presented in Table 3.

4.1.3Classical ML classifiers + BoW

In this context, the standalone CatBoost classifier exhibits superior performance compared to other single and ensemble methods across accuracy, precision, recall, and MCC metrics. However, in terms of AUC, F1, and Kappa, CatBoost ranks second. Among ensemble methods, the combination of CatBoost and XGBoost with SVM and Ridge, respectively, appears to surpass not only other ensembles but also the majority of individual methods. Once again, ensembles demonstrate a prevailing trend of better performance. Yet, it is important to acknowledge that the individual CatBoost method consistently achieves the highest scores in the majority of metrics. Table 4 presents the full results concerning this scenario.

4.1.4Classical ML Classifiers + TF-IDF

In this particular context, the standalone LR method claims the top position across all metrics except AUC and F1. Among the individual methods, CatBoost and Ridge follow in the hierarchy after LR. Turning to ensemble methods, the combination of CatBoost+LR+SVM emerges as the standout, surpassing all other ensembles. While LR showcases prominence in this scenario, it’s evident that ensembles consistently display superior performance compared to standalone methods. Notably, only a handful of individual methods within each metric manage to surpass the weakest-performing ensemble, underscoring the advantage of employing ensembles. The corresponding results can be found in Table 5.

4.1.5Deep learning pre-trained models

Regarding Deep Learning Pre-Trained models, RoBERTa exhibits a remarkable performance, outperforming all other models across all metrics. The BERT model follows in second place, while FiancialBERT and FinBERT (Yang & Hung) secure the third and fourth positions, respectively. FinBERT (Araci) can be found in the last place, considering all metrics. The indisputable dominance of RoBERTa is evident, as it consistently achieves superior scores, surpassing competitors by margins ranging from 2% to 7% in different metrics, as indicated in Table 6. It’s worth noting that ROC-AUC values for the FinBERT models could not be obtained due to the absence of necessary class probabilities from these models, which are crucial for calculating this metric. The corresponding results for all metrics are also depicted with a graphical representation of them, which can be found in Fig. 3. The AUC metric is excluded, as it could not be calculated for FinBERT models, due model limitations.

Figure 3.

Deep learning pre-trained models’ performance.

4.1.6General discussion

The design of the experimental procedure was thoughtfully structured to offer valuable insights into several distinct inquiries. The primary focus revolves around comparing the utilization of Deep Learning Pre-Trained Models and classic ML models, with a particular emphasis on assessing whether integrating more sophisticated models translates to improved performance. Within the domain of traditional ML models, two pivotal questions come to the forefront.

Firstly, there is the question of choosing between BoW and TF-IDF techniques, aiming to discern which among them yields more favorable outcomes. This inquiry directly concerns word representation, investigating which technique better captures the nuances of language and ultimately contributes to superior model performance.

Secondly, an additional aspect under scrutiny is the potential benefit derived from employing the SMOTE technique to artificially augment our data in order to tackle the problem of the initial imbalanced dataset. This question delves into the interplay between model performance and the integration of SMOTE, seeking to uncover whether the introduction of this technique positively impacts the final outcomes of the models.

Through a comprehensive exploration of these questions, the experimental design seeks to shed light on the intricate dynamics between various modelling approaches, offering nuanced insights into the strengths and limitations of each technique and aiding in informed decision-making for optimal model selection. To give a better insight into the results, in Fig. 4, the results of the five single best methods (according to MCC) are given in a graphical representation.

Figure 4.

Best 5 single classic ML methods.

Regarding the initial question, the results do not provide a definitive answer. Concerning the utilization of BoW or TF-IDF (when SMOTE is not applied), it appears that in certain methods, such as Catboost and XGBoost, BoW exhibits superior performance, while in others, TF-IDF proves to be more effective. However, when SMOTE is applied, TF-IDF outperforms BoW, notably enhancing performance across all metrics.

Turning to the second question, which pertains to the potential advantages of utilizing SMOTE, the answer becomes quite evident. When employing BoW, using SMOTE tends to lead to a reduction in model performance across all metrics. In certain instances, such as when employing the Ridge Classifier, this decline in performance is notably significant. On the other hand, when TF-IDF is employed, the results provide a clear picture. Specifically, in terms of Precision, F1 Score, Kappa, and MCC, the outcomes unambiguously indicate that SMOTE enhances model performance. However, when considering metrics like accuracy, AUC, and Recall, the situation becomes less straightforward. While in the majority of cases, the utilization of SMOTE proves beneficial, there are instances where its application appears to have a minor detrimental effect on model performance in these three metrics.

Regarding the utilization of ensemble techniques, it is important to note two key observations. First and foremost, ensembles generally tend to outperform the vast majority of individual ML models. However, it is worth noting that they do not surpass the best-performing single method in each specific scenario, except the AUC metric, where ensembles consistently outperform single methods across all cases. Our findings confirm the overall strong performance of ensembles. However, in the context of our study, their contribution is not particularly significant, and they may not be the preferred choice. It is worth mentioning that the tested ensembles were soft voting schemes. More sophisticated ensemble techniques may have the potential to be more powerful and possibly yield better results.

As we arrive at the final point of our findings, which pertains to the overall performance of classical ML classification algorithms compared to sophisticated Deep Learning Pre-Trained models, the outcome appears quite promising. Deep Learning Pre-Trained models consistently outperform their classic counterparts across all metrics. To be more specific, regarding the best-performing pre-trained model, RoBERTa, and the top single model from each of the four classical ML scenarios, RoBERTa showcases a substantial improvement in every metric. For example, RoBERTa achieves a 7% higher accuracy compared to the following method, which combines Logistic Regression (LR) with TF-IDF. In terms of the MCC metric, the results are even more compelling, with RoBERTa once again outperforming its closest rival by a margin of 17%. In Fig. 5, the corresponding results of all metrics are depicted.

Figure 5.

RoBERTa vs the best single classifier of each scenario.

However, it is worth noting an unexpected yet valuable outcome: both RoBERTa and BERT outperform the three variants of FinBERT. This discovery raises intriguing questions and warrants further investigation, especially considering that FinBERT models are fine-tuned specifically for financial tasks.