|
ИСТИНА |
Войти в систему Регистрация |
ИСТИНА ФИЦ ПХФ и МХ РАН |
||
Automated Detection of Human Values in Texts: ML Challenges and Performance Benchmarksстатья
- Авторы: Rink Olga, Maysuradze Archil, Fedorov Artem, Ischenko Roman, Korchagina Anna, Tabachenkov Andrey, Tsybanov Ilya, Vorontsov Konstantin
- Сборник: Lecture Notes in Computer Science
- Год издания: 2025
- Издательство: Springer Nature
- Местоположение издательства: Switzerland
- Первая страница: 304
- Последняя страница: 321
- DOI: 10.1007/978-3-031-93536-7_22
- Аннотация: Automated content analysis involves applying AI to insufficiently formalized areas of social studies and humanities. In 2023–2024, we researched the landscape of human values, represented in Russian social media writings, compiled a classifier, and labeled the collected data by 105 human values and their tonality. This paper discusses our development pipeline for generalized text markup structures, the challenges of machine learning to automatically identify human values in texts, own benchmark performance, the Human Values dataset in Russian, and evaluation procedures. The data set is used to create a RoBERTa-based model and a prompt-based model with Qwen 2.5 14B Instruct. These new models automatically categorize texts and focus on multi-annotator assessments. Benchmarks are created to compare the performance of these ML models with human annotators and a Mixture of Expects strategy. Both models demonstrate relatively high performance in detecting groups of human values, and sentiment classification does not significantly degrade it. The RoBERTa-based model exhibits a higher average performance (F1 0.753) in detecting parent classes in our hierarchical classifier compared to the Qwen-based model (F1 0.739); both indicate a good balance between precision and recall. In recognizing human values at child levels in the classifier RoBERTa (F1 0.274) performs significantly less effectively than the prompt-based strategy (F1 0.537). The dependence of the F1 score, obtained in the validation stage, on the number of annotated instances per class, used in training the RoBERTa-based model, demonstrates an almost linear relationship between the performance achieved and the number of annotated instances.
- Добавил в систему: Ринк Ольга Леонидовна