On readability evaluation of multimodal texts from Russian Universities websites (based on the PolyLing corpus)

M. S. Kogan, D. A. Gavrilik, S. V. Chistyakova, A. V. Rubtsova, E. R. Nikulina, A. V. Cherkas, M. V. Bolsunovskaya

Abstract


University websites being complex multimodal structures are very important in modern educational environment: being the central element of the University’s electronic information and educational environment, they play part of intermediary between University and the outer world and a powerful University image-maker’s tool. Multimodal university websites draw researchers’ attention since their appearance in the 1990s. However, the analysis of studies on University websites has revealed that very few of them focused on the websites’ readability. This paper focusing on evaluating readability of leading Russian university websites aims to partially bridge this gap. To achieve the goal, at the first stage of the project a corpus of 1000+ texts was built by parsing news sections of pre-selected websites. The automatic cluster analysis was used to determine the main themes of the built corpus named PolyLing. The texts’ readability was evaluated both automatically with Python script based on classical readability indices and by 132 human assessors who evaluated a representative sample of the corpus according to 10 criteria belonging to three groups (linguistic, structural and logical, and appealing) through completing a specially- designed questionnaire. The correlation analysis showed a satisfactory agreement for texts of middle difficulty and a negative correlation for easier and more difficult texts between the automatic and respondents’ estimates. The paper proposes possible explanation for this divergence and outlines further research.

Full Text:

PDF (Russian)

References


Nikulina E.R., Cherkas A.V., Kozina E.D., Boiko A.V., Dmitrieva L.A. Development of a service for text readability assessment via machine learning technologies // Sistemnyj analiz v proektirovanii i upravlenii: sbornik nauchnyh trudov / Trudy XXVI Mezhdunarodnoj nauchno-prakticheskoj konferencii «Sistemnyj analiz v proektirovanii i upravlenii», Sankt-Peterburg, October 13-14, 2022. SPb: POLITEH-PRESS, 2023. In 3 volumes. Vol. 2. P.232–240. DOI:10.18720/SPBPU/2/id23-103.

Tomášková R. A Walk through the Multimodal Landscape of University Websites // Brno Studies in English. 2015. Vol. 41. Iss. 1. P. 77–100.

Middleton I., McConnell M., Davidson G. Presenting a model for the structure and content of a university World Wide Web site //Journal of Information Science. 1999. Vol. 25. Iss. 3. P. 219-227.

Zollo S. A. Internationalization and Globalization. A Multimodal Analysis of Italian Universities’ Websites // Journal of Multimodal Communication Studies. 2016. Vol.3. Iss. 1-2. P.1-17.

Zhang Z.C., Tu W. Representation of international students at Australian university websites: A critical multimodal discourse analysis // Iberica. 2019. Vol. 37. P. 221–243. https://revistaiberica.org/index.php/iberica/article/view/116 (accessed date: 10.04.2023)

M.S. Kogan, D.A. Gavrilik, S.V. Chistyakova, A.V. Rubtsova, E.R. Nikulina, A.V. Cherkas, M.V. Bolsunovskaya

Zhang S., Tan S., Wignell P., O'Halloran K. Addressing international students on Australian and Chinese university webpages: A comparative study //Discourse, Context & Media. 2020. Vol.36. DOI: 10.1016/j.dcm.2020.100403.

Zhang Z., Tan S., O’Halloran K.L. Managing higher education and neoliberal marketing discourses on Why Choose webpages for international students on Australian and British university websites // Discourse & Communicationю 2022. Vol. 16. Iss. 4. P. 462–481. DOI: 10.1177/17504813221074076.

Chernyavskaya V.E., Zharkynbekova S.K. Linguistic and social construction of national university identity: Kazakh and Russian universities’ mission statements // Vestnik of Saint Petersburg University. Language and Literature. 2019. Vol. 16. Iss. 2. P. 304-319. DOI: 10.21638/spbu09.2019.210.

Chernyavskaya V.E., Safronenkova E.L. Towards constructing identity of a National University: “Our past” at the websites of Russian universities // J. Sib. Fed. Univ. Humanit. Soc. Sci. 2019. Vol. 12. Iss. 10. P. 1819–1839. DOI: 10.17516/1997-1370-0491.

Michelson K., Alvarez Valencia J.A. Study Abroad: Tourism or Education? A Multimodal Social Semiotic Analysis of Institutional Discourses of a Promotional Website // Discourse & Communication. 2016. Vol. 10, Iss. 3. P. 235–256. DOI: 10.1177/1750481315623893.

Hyland K. The presentation of self in scholarly life: Identity and marginalization in academic homepages // English for Specific Purposes. 2011. Vol. 30. Iss. 4. P. 286–297. DOI: 10.1016/j.esp.2011.04.004.

Bernardini S., Ferraresi A. The academic Web-as-Corpus // Proc. 8th Web as Corpus Workshop. Stroudsburg, PA: ACM. 2013. P. 53–62.

Bernardini S., Ferraresi A. Institutional academic English and its phraseology: native and lingua franca perspectives // English for Academic Purposes: Approaches and Implications / G. Diani, P. Thompson (eds.). Cambridge Scholars Publishing, Newcastle, UK. 2015. ch.9, P. 225 – 244.

Venuti M., Nasti C. Italian and UK university websites: comparing communicative strategies // ESP Across Cultures. 2015. Vol. 12. P. 127-137.

Nasti C., Venuti M., Zollo S.A. UK university websites: A multimodal, corpus-based analysis // International Journal of Language Studies. 2017. Vol. 11. Iss. 4. P. 131–152.

Reynolds R.J. Russian natural language processing for computer-assisted language learning. PhD dissertation, UiT. Tromsø: The Arctic University of Norway. 2016. https://munin.uit.no/bitstream/handle/10037/9685/thesis.pdf?sequence=3&isAllowed=y (accessed date: 10.04.2023).

Solnyshkina M.I.,. Kiselnikov A.S. Text complexity: study phases in Russian linguistics // Tomsk State University Journal of Philology. 2015. Vol. 38. No. 6. P. 86–99. DOI: 10.17223/19986645/38/7.

Lyashevskaya O. K opredeleniju slozhnosti russkih tekstov // XVII Aprel'skaja mezhdunarodnaja nauchnaja konferencija po problemam razvitija jekonomiki i obshhestva: v 4 kn. / E. G. Jasin (ed.) ; National research university “Higher School of Economics”. Moscow : Publishing house of HSE, 2017. Kniga 4. P. 408–418. https://conf.hse.ru/data/2017/04/06/1168267884/XVII%20%D0%90%D0%9C%D0%9D%D0%9A_%D0%9A%D0%BD.4-%D1%81%D0%B0%D0%B9%D1%82.pdf (accessed date: 10.04.2023).

Laposhina A.N. Insights from an experimental study on the text complexity for Russian as a foreign language // Dinamika jazykovyh i kul'turnyh processov v sovremennoj Rossii: Proceedings of VI Congress ROPRYAL (Ufa, Oct. 11-14, 2018). 2018. Iss. 6. P. 1154-1179.

Solovyev V., Solnyshkina M., Ivanov V. Prediction of reading difficulty in Russian academic texts // Journal оf Intelligent & Fuzzy Systems. 2019. Vol. 36. Iss. 5. P. 4553-4563. DOI: 10.3233/JIFS-179007.

Solovyev V., Ivanov V., Solnyshkina M. Assessment of reading difficulty levels in Russian academic texts: Approaches and metrics // Journal of Intelligent & Fuzzy Systems. 2018. Vol. 34. Iss. 5. P. 3049–3058. DOI: 10.3233/JIFS-169489.

Blinova О.V., Tarasov N.A. Complexity of Russian legal texts: assessment methods and language data // Proc. International Conference “Corpora 2021” (July 1-3, 2021). Skifija-print, Saint Petersburg. 2021. P. 175–182.

Saveliev D.A. A study in complexity of sentences constituting Russian Federation legal acts // Law. Journal of HSE. 2020. No. 1. P. 50–74. DOI: 10.17323/2072-8166.2020.1.50.74.

Akgül Y. Evaluating the performance of websites from a public value, usability, and readability perspectives: a review of Turkish national government websites // Univ Access Inf Soc. Published online Aug. 2022. DOI: 10.1007/s10209-022-00909-4.

Akgül Y. The Accessibility, Usability, Quality and Readability of Turkish State and Local Government Websites: an Exploratory Study // International Journal of Electronic Government Research. 2019. Vol. 15. Iss. 1. P. 62–81. DOI:10.4018/IJEGR.2019010105.

Karhu M., Hilera J.R., Fernández L., Ríos R. Accessibility and readability of university websites in Finland // J. of Accessibility and. Design for All. 2012. Vol. 2. Iss. 2. P. 178–189. DOI: 10.17411/jacces.v2i2.70.

Patra M.R., Dash A.R., Mishra P.K. A quantitative analysis of WCAG 2.0 compliance for some Indian web portals // International Journal of Computer Science, Engineering and Applications (IJCSEA). 2017. Vol. 4. Iss. 1. P. 9–23. DOI: 10.48550/arXiv.1710.08788.

Akgül Y. Accessibility, usability, quality performance, and readability evaluation of university websites of Turkey: a comparative study of state and private universities // Univ Access Inf Soc. 2021. Vol.20. Iss. 1. P. 157–170. DOI: 10.1007/s10209-020-00715-w.

Rashida M., Islam K., M. Kayes A.S., Hammoudeh M., Arefin M.S., Habib M.A. Towards developing a framework to analyze the qualities of the university websites // Computers. 2021.Vol. 10. Iss. 57. P. 1–16. DOI: 10.3390/computers 10050057.

Svidetel'stvo № 2022669940. Programma dlya avtomaticheskogo sbora tekstov s novostnyh rubrik sajtov vuzov: № 2022668973: zayavl. 17.10.2022: opubl. 26.10.2022 / Rakova V.V., Cherkas A.V., Kozina E.D., Rubtsova A.V., Kogan M.S. 1p.

Korobov M. Morphological analyzer and generator for Russian and Ukrainian languages // M. Khachay, N. Konstantinova, A. Panchenko, D. Ignatov, V. Labunets (eds). Analysis of Images, Social Networks and Texts. AIST 2015 / Communications in Computer and Information Science. Springer, Cham. 2015. Vol. 542. Р. 320–332. DOI: 10.1007/978-3-319-26123-2_31.

Jain A. K. Data clustering: 50 years beyond K-means // Pattern Recognition Letters. 2010. Vol. 31. Iss. 8. P. 651–666. DOI: 10.1016/j.patrec.2009.09.011.

Rousseeuw P.J. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis // Computational and Applied Mathematics. 1987. Vol. 20. P. 53-65.

Gómez P.C., Sánchez-Lafuente Á.A. Readability indices for the assessment of textbooks: a feasibility study in the context of EFL // Vigo Intern. J. of Appl. Linguistics (VIAL). 2019. Vol. 16. P. 31–52.

Ivanov V., Solnyshkina M., Solovyev V. Efficiency of text readability features in Russian academic texts // Computational Linguistics and Intellectual Technologies. Proc. Intern. Conf. “Dialogue 21” (May 30 – June 21, 2018). Moscow, Russia. 2018. P. 284–293. http://www.dialog-21.ru/media/4302/ivanovvv.pdf (accessed date: 10.04.2023)

Blinova O., Tarasov N. Complexity metrics of Russian legal texts: selection, use, initial efficiency evaluation // Computational Linguistics and Intellectual Technologies: Proc. International Conference “Dialogue 2022” (June 15 – 18, 2022). Moscow. 2022. P. 1017-1028. DOI: 10.28995/2075-7182-2022-21-1017-1028.

Koltsova O.Yu, Alexeeva S.V., Kolcov S.N. An Opinion word lexicon and a training data set for Russian sentiment analysis of social media computational linguistics and intellectual technologies // Proc. Intern. Conf. “Dialogue 2016” (June 1 – 4, 2016). Moscow, Russia. 2016. P. 277–287. URL: https://www.dialog-21.ru/digest/2016/articles/ (accessed date: 10.04.2023).


Refbacks

  • There are currently no refbacks.


Abava  Кибербезопасность IT Congress 2024

ISSN: 2307-8162