Using Data Mining in the Sentiment Analysis Process on the Financial Market

Marian Pompiliu Cristescu; Raluca Andreea Nerişanu; Dumitru Alexandru Mara

doi:10.2478/jses-2022-0003

Abstract

Sentiment analysis refers to the analysis of human opinions and sentiments that are expressed in written text, being also a part of the Natural Language Processing (NLP) tasks. Sentiment analysis can be applied in different domains, especially in the corporate marketing and sales, the healthcare system or the financial market analysis. In this paper we aim to highlight how data mining is able to extract the sentiment score from a financial platform that shows the major headlines regarding stocks, in order to highlight the publications’ positive or negative opinion over a stock. In order to gain the sentiment score we have scraped text data from the platform Finviz from which the polarity of the opinion may be extracted. We have also used Valence Aware Dictionary for Sentiment Reasoning (VADER), by running a Python script using the BeautifulSoup library. After that we have used Pandas (Python Data Analysis Library) to analyse and obtain a sentiment score on the article headlines. Results show that the script is able to generate the sentiment score for various selected stocks, while also showing graphical diagrams for the past and future trend of the stock, in terms of overall opinion on the stock performance.

References

Al-Shabi, M. (2020). Evaluating the performance of the most important Lexicons used to Sentiment analysis and opinions Mining, IJCSNS International Journal of Computer Science and Network Security, 20(1), January 2020.
Search in Google Scholar Back to article
Assunção, M. D. et al. (2015). Big Data computing and clouds: Trends and future directions, Journal of Parallel and Distributed Computing, 79–80, pp.3–15. doi: 10.1016/j.jpdc.2014.08.003.10.1016/j.jpdc.2014.08.003
Search in Google Scholar Back to article
Balazs, J. A. and Velásquez, J. D. (2016). Opinion Mining and Information Fusion: A survey, Information Fusion, 27, pp. 95–110. doi: 10.1016/j.inffus.2015.06.002.10.1016/j.inffus.2015.06.002
Search in Google Scholar Back to article
Barham, H. (2017). Achieving competitive advantage through big data: A literature review, PICMET 2017 - Portland International Conference on Management of Engineering and Technology: Technology Management for the Interconnected World, Proceedings, 2017-Janua, p. 1–7. doi: 10.23919/PICMET.2017.8125459.10.23919/PICMET.2017.8125459
Search in Google Scholar Back to article
Berger, A. L., Della Pietra, S. A. and Della Pietra, V. J. (1996). A Maximum Entropy Approach to Natural Language Processing, Computational Linguistice, Cambridge, MA: MIT Press, 22(1), p. 39–71. Available at: https://aclanthology.org/J96-1002.
Search in Google Scholar Back to article
Chaturvedi, I. et al. (2018). Distinguishing between facts and opinions for sentiment analysis: Survey and challenges, Information Fusion, 44, p. 65–77. doi: 10.1016/j.inffus.2017.12.006.10.1016/j.inffus.2017.12.006
Search in Google Scholar Back to article
Colasanto, F. et al. (2022). BERT’s sentiment score for portfolio optimization: a fine-tuned view in Black and Litterman model, Neural Computing and Applications. Springer London, 1. doi: 10.1007/s00521-022-07403-1.10.1007/s00521-022-07403-1
Search in Google Scholar Back to article
Cordeiro, E. R. et al. (2014). Posttherapy Follow-up and First Intervention, Prostate Cancer: Diagnosis and Clinical Management, (June), pp. 211–229. doi: 10.1002/9781118347379.ch11. Cortes, C. and Vapnik, V. (1995). ‘Support-vector networks’, Machine Learning, 20(3), p. 273–297. doi: 10.1007/BF00994018.10.1007/BF00994018
Search in Google Scholar Back to article
Denecke, K. (2008). Using SentiWordNet for multilingual sentiment analysis, in 2008 IEEE 24th International Conference on Data Engineering Workshop. IEEE, pp. 507–512. doi: 10.1109/ICDEW.2008.4498370.10.1109/ICDEW.2008.4498370
Search in Google Scholar Back to article
Devlin, J. et al. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding, NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 1(Mlm), pp. 4171–4186.
Search in Google Scholar Back to article
Erevelles, S., Fukawa, N. and Swayne, L. (2016). Big Data consumer analytics and the transformation of marketing, Journal of Business Research, 69(2), p. 897–904. doi: 10.1016/j.jbusres.2015.07.001.10.1016/j.jbusres.2015.07.001
Search in Google Scholar Back to article
Ertel, W. (2017). Machine Learning and Data Mining, in, pp. 175–243. doi: 10.1007/978-3-319-58487-4_8.10.1007/978-3-319-58487-4_8
Search in Google Scholar Back to article
Esuli, A. and Sebastiani, F. (2006). {SENTIWORDNET}: A Publicly Available Lexical Resource for Opinion Mining, in Proceedings of the Fifth International Conference on Language Resources and Evaluation ({LREC}{’}06). Genoa, Italy: European Language Resources Association (ELRA). Available at: http://www.lrecconf.org/proceedings/lrec2006/pdf/384_pdf.pdf.
Search in Google Scholar Back to article
Fan, J. and Gu, J. (2003). Semiparametric estimation of Value at Risk, The Econometrics Journal, 6(2), pp. 261–290. doi: 10.1111/1368-423X.t01-1-00109.10.1111/1368-423X.t01-1-00109
Search in Google Scholar Back to article
Gandomi, A. and Haider, M. (2015). Beyond the hype: Big data concepts, methods, and analytics, International Journal of Information Management, 35(2), p. 137–144. doi: 10.1016/j.ijinfomgt.2014.10.007.10.1016/j.ijinfomgt.2014.10.007
Search in Google Scholar Back to article
Gautam, G. and Yadav, D. (2014). Sentiment analysis of twitter data using machine learning approaches and semantic analysis, in 2014 Seventh International Conference on Contemporary Computing (IC3). IEEE, p. 437–442. doi: 10.1109/IC3.2014.6897213.10.1109/IC3.2014.6897213
Search in Google Scholar Back to article
Giambattista, Amati Giuseppe, A. et al. (2008). FUB, IASI-CNR and University of Tor Vergata at TREC 2008 Blog Track, NIST Special Publication.10.6028/NIST.SP.500-277.blog-fub
Search in Google Scholar Back to article
Giudici, P., Mezzetti, M. and Muliere, P. (2003). Mixtures of products of Dirichlet process for variable selection in survival analysis, Journal of Statistical Planning and Inference, 111(1–2), p. 101–115. doi: 10.1016/S0378-3758(02)00291-4.10.1016/S0378-3758(02)00291-4
Search in Google Scholar Back to article
Godbole, N., Manjunath, S. and Skiena, S. (2007). Large-Scale Sentiment Analysis for News and Blogs Namrata, in Conference: Proceedings of the International Conference on Weblogs and Social Media.
Search in Google Scholar Back to article
Hemmatian, F. and Sohrabi, M. K. (2019). A survey on classification techniques for opinion mining and sentiment analysis, Artificial Intelligence Review, 52(3), pp. 1495–1545. doi: 10.1007/s10462-017-9599-6.10.1007/s10462-017-9599-6
Search in Google Scholar Back to article
Hu, M. and Liu, B. (2004). Mining and summarizing customer reviews, KDD-2004 - Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 168–177. doi: 10.1145/1014052.1014073.10.1145/1014052.1014073
Search in Google Scholar Back to article
Hutto, C.J. and Gilbert, E. (2014). VADER: A Parsimonious Rule-based Model for, Eighth International AAAI Conference on Weblogs and Social Media, pp.18. Available at: https://www.aaai.org/ocs/index.php/ICWSM/ICWSM14/paper/viewPaper/8109.
Search in Google Scholar Back to article
Kabir, N. and Carayannis, E. (2013). Big data, tacit knowledge and organizational competitiveness, Journal of Intelligence Studies in Business, 3(3), pp.54–62. doi: 10.37380/jisib.v3i3.76.10.37380/jisib.v3i3.76
Search in Google Scholar Back to article
Karimi, A., Rossi, L. and Prati, A. (2021). AEDA: An Easier Data Augmentation Technique for Text Classification, Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021, pp. 2748–2754. doi: 10.18653/v1/2021.findings-emnlp.234.10.18653/v1/2021.findings-emnlp.234
Search in Google Scholar Back to article
Li, X. et al. (2019). Exploiting bert for end-to-end aspect-based sentiment analysis_, WNUT@EMNLP 2019 - 5th Workshop on Noisy User-Generated Text, Proceedings, pp. 34–41. doi: 10.18653/v1/d19-5505.10.18653/v1/D19-5505
Search in Google Scholar Back to article
Liu, B. et al. (1998). Integrating Classification and Association Rule Mining, Knowledge Discovery and Data Mining, pp.80–86. Available at: http://www.aaai.org/Papers/KDD/1998/KDD98-012.pdf%5Cnhttp://www.aaai.org/Library/KDD/1998/kdd98-012.php%5Cnhttp://citeseer.ist.psu.edu/liu98integrating.html.
Search in Google Scholar Back to article
Liu, Y. et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach, (1). Available at: http://arxiv.org/abs/1907.11692.
Search in Google Scholar Back to article
Ma, D. et al. (2017). Interactive Attention Networks for Aspect-Level Sentiment Classification, in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. California: International Joint Conferences on Artificial Intelligence Organization, pp. 4068–4074. doi: 10.24963/ijcai.2017/568.10.24963/ijcai.2017/568
Search in Google Scholar Back to article
Manyika, J. et al. (2011). Big data: The next frontier for innovation, competition and productivity, McKinsey Global Institute, (June), pp.156. Available at: https://bigdatawg.nist.gov/pdf/MGI_big_data_full_report.pdf.
Search in Google Scholar Back to article
Michael Steinbach, George Karypis and Vipin Kumar (2000). A Comparison of Document Clustering Techniques, KDD workshop on text mining, pp.1–2. Available at: https://www.bibsonomy.org/bibtex/210e5c1e3ff54d9dce505a231f8ae7b32/hotho.
Search in Google Scholar Back to article
Miller, G. A. (1995). WordNet: A Lexical Database for English, Communications of the ACM, 38(11), pp.39–41. doi: 10.1145/219717.219748.10.1145/219717.219748
Search in Google Scholar Back to article
Mohanty, A. K., Senapati, M. R. and Lenka, S. K. (2013). An improved data mining technique for classification and detection of breast cancer from mammograms, Neural Computing and Applications, 22(1), pp.303–310. doi: 10.1007/s00521-012-0834-4.10.1007/s00521-012-0834-4
Search in Google Scholar Back to article
Phan, M. H. and Ogunbona, P. O. (2020). Modelling Context and Syntactical Features for Aspect-based Sentiment Analysis, pp. 3211–3220. doi: 10.18653/v1/2020.acl-main.293.10.18653/v1/2020.acl-main.293
Search in Google Scholar Back to article
Prathi, J. K., Raparthi, P. K. and Gopalachari, M. V. (2020). Real-Time Aspect-Based Sentiment Analysis on Consumer Reviews, Data Engineering and Communication Technology. Advances in Intelligent Systems and Computing, pp. 801–810. doi: 10.1007/978-981-15-1097-7_67.10.1007/978-981-15-1097-7_67
Search in Google Scholar Back to article
Provost, F. and Fawcett, T. (2013). Data Science and its Relationship to Big Data and Data-Driven Decision Making, Big Data, 1(1), pp. 51–59. doi: 10.1089/big.2013.1508.10.1089/big.2013.1508
Search in Google Scholar Back to article
Russom, P. (2021) Big data analytics, A Closer Look at Big Data Analytics.
Search in Google Scholar Back to article
Sangeetha, S. and Sreeja, A. K. (2015). No Science No Humans, No New Technologies No Changes “Big Data a Great Revolution”, International Journal of Computer Science and Information Technologies, 6(4), pp. 3269–3274.
Search in Google Scholar Back to article
Sheela, L. J. (2016). A Review of Sentiment Analysis in Twitter Data Using Hadoop, International Journal of Database Theory and Application, 9(1), pp. 77–86. doi: 10.14257/ijdta.2016.9.1.07.10.14257/ijdta.2016.9.1.07
Search in Google Scholar Back to article
Shim, J. P. et al. (2015). Big data and analytics: Issues, solutions, and ROI, Communications of the Association for Information Systems, 37(1), pp. 797–810. doi: 10.17705/1cais.03739.10.17705/1CAIS.03739
Search in Google Scholar Back to article
Siganos, A., Vagenas-Nanos, E. and Verwijmeren, P. (2017). Divergence of sentiment and stock market trading, Journal of Banking & Finance, 78, pp. 130–141. doi: 10.1016/j.jbankfin.2017.02.005.10.1016/j.jbankfin.2017.02.005
Search in Google Scholar Back to article
Thelwall, M., Homsi, M. N. and Prabowo, R. (2009). Sentiment analysis: A combined approach Cite this paper Related papers SA2 vinodhini Manieniyan Sent iment Analysis and Sent iment Classificat ion using NLP IRJET Journal Mult i-Class Sent iment Analysis using a Hierarchical Logist ic Model Tree Approach.
Search in Google Scholar Back to article
Theodorou, T. I. et al. (2021). An AI-enabled stock prediction platform combining news and social sensing with financial statements, Future Internet, 13(6), pp. 1–22. doi: 10.3390/fi13060138.10.3390/fi13060138
Search in Google Scholar Back to article
Wei, Y.-C. et al. (2017). Informativeness of the market news sentiment in the Taiwan stock market, The North American Journal of Economics and Finance, 39, pp. 158–181. doi: 10.1016/j.najef.2016.10.004.10.1016/j.najef.2016.10.004
Search in Google Scholar Back to article
Weng, B., Ahmed, M. A. and Megahed, F. M. (2017). Stock market one-day ahead movement prediction using disparate data sources, Expert Systems with Applications, 79, pp. 153–163. doi: 10.1016/j.eswa.2017.02.041.10.1016/j.eswa.2017.02.041
Search in Google Scholar Back to article
Wu, X. et al. (2019). Conditional BERT Contextual Augmentation, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11539 LNCS, pp.84–95. doi: 10.1007/978-3-030-22747-0_7.10.1007/978-3-030-22747-0_7
Search in Google Scholar Back to article
Xu, C. et al. (2020). BERT-of-Theseus: Compressing BERT by Progressive Module Replacing, in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA, USA: Association for Computational Linguistics, p.7859–7869. doi: 10.18653/v1/2020.emnlp-main.633.10.18653/v1/2020.emnlp-main.633
Search in Google Scholar Back to article
Xu, H. et al. (2020). DomBERT: Domain-oriented language model for aspect-based sentiment analysis, Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020, pp.1725–1731. doi: 10.18653/v1/2020.findings-emnlp.156.10.18653/v1/2020.findings-emnlp.156
Search in Google Scholar Back to article
Xu, H. et al. (2021). Understanding Pre-trained BERT for Aspect-based Sentiment Analysis, p.244–250. doi: 10.18653/v1/2020.coling-main.21.10.18653/v1/2020.coling-main.21
Search in Google Scholar Back to article
Yi, J. et al. (2003). Sentiment analyzer: extracting sentiments about a given topic using natural language processing techniques, in Third IEEE International Conference on Data Mining. IEEE Comput. Soc, pp. 427–434. doi: 10.1109/ICDM.2003.1250949.10.1109/ICDM.2003.1250949
Search in Google Scholar Back to article
Yıldırım, D. C., Toroslu, I. H. and Fiore, U. (2021). Forecasting directional movement of Forex data using LSTM with technical and macroeconomic indicators, Financial Innovation. Springer Berlin Heidelberg, 7(1), pp. 1–36. doi: 10.1186/s40854-020-00220-2.10.1186/s40854-020-00220-2
Search in Google Scholar Back to article
Zhang, X. et al. (2018). Improving stock market prediction via heterogeneous information fusion, Knowledge-Based Systems, 143, pp. 236–247. doi: 10.1016/j.knosys.2017.12.025.10.1016/j.knosys.2017.12.025
Search in Google Scholar Back to article

Using Data Mining in the Sentiment Analysis Process on the Financial Market

Abstract

Paradigm

My account