System Approach to the Combined Use of Large Language Models and Classical Models in Foresight Tasks
DOI:
https://doi.org/10.20535/kpisn.2024.1-4.315079Keywords:
system analysis; foresight; textual analytics; classification; LLM; NLPAbstract
Background. Large Language Models (LLMs) and their associated agents have spread wide technology and represent a significant advancement in recent times. These state-of-the-art models expose valuable potential, but they are not devoid of restrictions, inefficiencies, and limits. This article investigates the exploration of these constraints within specific domain areas and prediction problems as examples.
Objective. The article highlights features offered by GPT-based models and compares the conclusions with classical methods of textual data analysis in classification tasks using the prediction methodology as an example. The purpose of the study is to develop a system approach to the combined use of traditional machine learning approaches as a practical alternative to LLMs in foresight tasks using the example of STEEP analysis, which provides an opportunity to obtain valuable information from textual data.
Methods. The study is structured into four segments, each addressing distinct parts: Data Mining, text pre-processing using LLMs, text pre-processing utilizing Natural Language Processing (NLP) methods, and comparative analysis of results. Data Mining includes data collection and data pre-processing stages for train and test observations. For the utilization of LLMs, chains of thought techniques and prompt engineering were used.
Results. Throughout this study, it was acknowledged that the LLMs can be used in combination with classical machine learning methodologies for domain-specific areas in STEEP analysis under Foresight tasks. The outcome revealed a model that was developed significantly faster and with less complexity compared to LLMs such as GPT and Mistral. Increasing the number of models employed leads to more stable results.
Conclusions. The main result of the proceeding is that the patterns that reveal LLMs under certain settings can also be identified by classical models. Moreover, augmenting the deployment of LLMs during the data preparation stages contributes to heightened stability in outcomes. Using classical models combined with LLMs speeds up response times during inference and reduces operating costs for running models.
References
Li et al. “Knowledge structure of technology licensing based on co-keywords network: A review and future directions” International Review of Economics & Finance, vol. 66, pp. 154–165, 2020. doi: 10.1016/j.iref.2019.11.007.
UNIDO, UNIDO Technology Foresight Manual, vol. 1. United Nations Industrial Development Organization, Vienna (2005).
European Commission. 2020. “Strategic Foresight Report – Charting the course towards a more resilient Europe”. Available online at: https://ec.europa.eu/info/strategy/priorities-2019-2024/new-push-european-democracy/strategic-foresight/2020-strategic-foresight-report_en.
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu and Michael Zeng, “Want To Reduce Labeling Cost? GPT-3 Can Help”, arXiv preprint arXiv.2108.13487, 2023, doi: 10.48550/arXiv.2108.13487.
P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing”, ACM Computing Surveys, vol. 55, no. 9, pp. 1–35, 2023.
A. B. Rosa, N. Gudowsky, P. Repo “Repo Sensemaking and lens-shaping: Identifying citizen contributions to foresight through comparative topic modelling”, Futures 129, pp. 1–15, 2021, doi: 10.1016/j.futures.2021.102733.
M. Vignoli, J. Rörden, D. Wasserbacher and S Kimpeler, “An Exploration of the Potential of Machine Learning Tools for Media Analysis to Support Sense-Making Processes in Foresight”, Frontiers in Communication 7:750614, 2022. doi :10.3389/fcomm.2022.750614.
N. Pankratova, V. Savastiyanov, “Assessment of situations in the field of social disasters based on the methodology of foresight and textual analytics”. Proceedings of the 2019 IEEE. Second International Conference IEEE UKRCON-2019. pp. 1207–1210, ISBN 9781728138831.
Callie Y. Kim, Christine P. Lee and Bilge Mutlu, “Understanding Large-Language Model (LLM)-powered Human-Robot Interaction”, In Proceedings of the 2024 ACM/IEEE, International Conference on Human-Robot Interaction (HRI ’24). Association for Computing Machinery, New York, NY, USA, pp. 371–380, 2024, doi: 10.1145/3610977.3634966.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
B. Zhang and H. Soh, “Large Language Models as Zero-Shot Human Models for Human-Robot Interaction”, 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA, pp. 7961–7968, 2023, doi: 10.1109/IROS55552.2023.10341488.
F. Pourpanah et al., “A Review of Generalized Zero-Shot Learning Methods”, in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4051–4070, 1 April 2023, doi: 10.1109/TPAMI.2022.3191696.
C. Mühlroth and M. Grottke, “Artificial Intelligence in Innovation: How to Spot Emerging Trends and Technologies”, in IEEE Transactions on Engineering Management, vol. 69, no. 2, pp. 493–510, April 2022, doi: 10.1109/TEM.2020.2989214.
M. Stoliar and V. Savastiyanov, “Using LLM classification in foresight studies”: Scientific Collection “InterConf”, vol. 157, pp. 367–375, 2023.
En Yu, Liang Zhao, Yana Wei, Jinrong Yang, Dongming Wu, Lingyu Kong, Haoran Wei, Tiancai Wang, Zheng Ge, Xiangyu Zhang and Wenbing Tao, “Merlin: Empowering Multimodal LLMs with Foresight Minds”, arXiv preprint arXiv:2312.00589, 2023, doi: 10.48550/arXiv.2312.00589.
Y. Luo, S. Xu and C. Xie, “E-commerce Big Data Classification and Mining Algorithm based on Artificial Intelligence”, 2022 IEEE 2nd International Conference on Electronic Technology, Communication and Information (ICETCI), Changchun, China, pp. 1153–1155, 2022, doi: 10.1109/ICETCI55101.2022.9832370.
Chan A., “GPT-3 and InstructGPT: technological dystopianism, utopianism, and “Contextual” perspectives in AI ethics and industry”, AI Ethics 3, pp. 53–64, 2023, doi: 10.1007/s43681-022-00148-6.
Dave Wild, “Futurework – A Guidebook for The Future of Work”, Aotearoa New Zealand: Smith & Wild. 218 p., 2023, ISBN 978-0-473-66594-4.
Mowafak Allaham and Nicholas Diakopoulos, “Supporting Anticipatory Governance using LLMs: Evaluating and Aligning Large Language Models with the News Media to Anticipate the Negative Impacts of AI”, arXiv preprint arXiv:2401.18028, 2024, doi: 10.48550/arXiv.2401.18028.
Cambria Erik, Bebo White., “Jumping NLP curves: A review of natural language processing research”, IEEE Computational intelligence magazine 9.2, pp. 48–57, 2014, doi: 10.1109/MCI.2014.2307227.
A. Jalilifard, V.F. Caridá, A.F. Mansano, R.S. Cristo, F.P.C. da Fonseca, “Semantic Sensitive TF-IDF to Determine Word Relevance in Documents”, Thampi, S.M., Gelenbe, E., Atiquzzaman, M., Chaudhary, V., Li, KC. (eds) Advances in Computing and Network Communications. Lecture Notes in Electrical Engineering, vol 736. Springer, Singapore, 2021, doi.org: 10.1007/978-981-33-6987-0_27.
B. Ahmed, G. Ali, A. Hussain, A. Baseer and J. Ahmed, “Analysis of Text Feature Extractors using Deep Learning on Fake News”, Eng. Technol. Appl. Sci. Res., vol. 11, no. 2, pp. 7001–7005, Apr. 2021.
Lucky Agarwal, Kartik Thakral, Gaurav Bhatt, and Ankush Mittal, “Authorship Clustering using TF-IDF weighted Word-Embeddings”, In Proceedings of the 11th Annual Meeting of the Forum for Information Retrieval Evaluation (FIRE ’19). Association for Computing Machinery, New York, NY, USA, pp. 24–29, 2019, doi: 10.1145/3368567.3368572.
Fanghui Liu, Xiaolin Huang, and Jie Yang, “Indefinite Kernel Logistic Regression”, In Proceedings of the 25th ACM International Conference on Multimedia (MM ’17). Association for Computing Machinery, New York, NY, USA, pp. 846–853, 2017, doi: 10.1145/3123266.3123295
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Volodymyr Savastiyanov, Mykhailo Stoliar
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under CC BY 4.0 that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work
