publications
publications in reversed chronological order.
2025
- Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto FrontierTheresia Veronika Rampisela, Tuukka Ruotsalo, Maria Maistro, and Christina LiomaIn Proceedings of the ACM on Web Conference 2025, Sydney NSW, Australia, 2025
Fairness and relevance are two important aspects of recommender systems (RSs). Typically, they are evaluated either (i) separately by individual measures of fairness and relevance, or (ii) jointly using a single measure that accounts for fairness with respect to relevance. However, approach (i) often does not provide a reliable joint estimate of the goodness of the models, as it has two different best models: one for fairness and another for relevance. Approach (ii) is also problematic because these measures tend to be ad-hoc and do not relate well to traditional relevance measures, like NDCG. Motivated by this, we present a new approach for jointly evaluating fairness and relevance in RSs: Distance to Pareto Frontier (DPFR). Given some user-item interaction data, we compute their Pareto frontier for a pair of existing relevance and fairness measures, and then use the distance from the frontier as a measure of the jointly achievable fairness and relevance. Our approach is modular and intuitive as it can be computed with existing measures. Experiments with 4 RS models, 3 re-ranking strategies, and 6 datasets show that existing metrics have inconsistent associations with our Pareto-optimal solution, making DPFR a more robust and theoretically well-founded joint measure for assessing fairness and relevance. Our code: https://github.com/theresiavr/DPFR-recsys-evaluation
@inproceedings{Rampisela2025JointFrontier, author = {Rampisela, Theresia Veronika and Ruotsalo, Tuukka and Maistro, Maria and Lioma, Christina}, title = {Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier}, year = {2025}, isbn = {9798400712746}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3696410.3714589}, doi = {10.1145/3696410.3714589}, booktitle = {Proceedings of the ACM on Web Conference 2025}, pages = {1548–1566}, numpages = {19}, keywords = {evaluation, fairness, pareto frontier, recommendation, relevance}, location = {Sydney NSW, Australia}, series = {WWW '25}, }
2024
- Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and RelevanceTheresia Veronika Rampisela, Tuukka Ruotsalo, Maria Maistro, and Christina LiomaIn Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Washington DC, USA, 2024
Relevance and fairness are two major objectives of recommender systems (RSs). Recent work proposes measures of RS fairness that are either independent from relevance (fairness-only) or conditioned on relevance (joint measures). While fairness-only measures have been studied extensively, we look into whether joint measures can be trusted. We collect all joint evaluation measures of RS relevance and fairness, and ask: How much do they agree with each other? To what extent do they agree with relevance/fairness measures? How sensitive are they to changes in rank position, or to increasingly fair and relevant recommendations? We eempirically study for the first time the behaviour of these measures across 4 real-world datasets and 4 recommenders. We find that most of these measures: i) correlate weakly with one another and even contradict each other at times; ii) are less sensitive to rank position changes than relevance- and fairness-only measures, meaning that they are less granular than traditional RS measures; and iii) tend to compress scores at the low end of their range, meaning that they are not very expressive. We counter the above limitations with a set of guidelines on the appropriate usage of such measures, i.e., they should be used with caution due to their tendency to contradict each other and of having a very small empirical range.
@inproceedings{Rampisela2024CanRelevance, author = {Rampisela, Theresia Veronika and Ruotsalo, Tuukka and Maistro, Maria and Lioma, Christina}, title = {Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance}, year = {2024}, isbn = {9798400704314}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3626772.3657832}, doi = {10.1145/3626772.3657832}, booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval}, pages = {271–281}, numpages = {11}, keywords = {fairness and relevance evaluation, recommender systems}, location = {Washington DC, USA}, series = {SIGIR '24}, } - Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical StudyTheresia Veronika Rampisela, Maria Maistro, Tuukka Ruotsalo, and Christina LiomaACM Trans. Recomm. Syst., Nov 2024
This paper has been awarded the “Women in RecSys” Journal Paper of the Year Award 2024 (Junior Category) at the RecSys 2024 conference. The award is given to women-authored journal papers that have distinguished themselves due to their innovativeness and scientific rigor.
Fairness is an emerging and challenging topic in recommender systems. In recent years, various ways of evaluating and therefore improving fairness have emerged. In this study, we examine existing evaluation measures of fairness in recommender systems. Specifically, we focus solely on exposure-based fairness measures of individual items that aim at quantifying the disparity in how individual items are recommended to users, separate from item relevance to users. We gather all such measures and we critically analyse their theoretical properties. We identify a series of limitations in each of them, which collectively may render the affected measures hard or impossible to interpret, to compute, or to use for comparing recommendations. We resolve these limitations by redefining or correcting the affected measures, or we argue why certain limitations cannot be resolved. We further perform a comprehensive empirical analysis of both the original and our corrected versions of these fairness measures, using real-world and synthetic datasets. Our analysis provides novel insights into the relationship between measures based on different fairness concepts, and different levels of measure sensitivity and strictness. We conclude with practical suggestions of which fairness measures should be used and when. Our code is publicly available. To our knowledge, this is the first critical comparison of individual item fairness measures in recommender systems.
@article{Rampisela2023EvaluationStudy, author = {Rampisela, Theresia Veronika and Maistro, Maria and Ruotsalo, Tuukka and Lioma, Christina}, title = {Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study}, year = {2024}, issue_date = {June 2025}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {3}, number = {2}, url = {https://doi.org/10.1145/3631943}, doi = {10.1145/3631943}, journal = {ACM Trans. Recomm. Syst.}, month = nov, articleno = {18}, numpages = {52}, keywords = {Item fairness, individual fairness, fairness measures, evaluation measures, recommender systems}, The award is given to women-authored journal papers that have distinguished themselves due to their innovativeness and scientific rigor. }, }
2021
- Classification of the likelihood of Indonesian Facebook users in spreading hoaxes using Support Vector Machine (SVM)Theresia Veronika Rampisela, Hanna Tiara Andarlia, and Zuherman RustamIn Journal of Physics: Conference Series, 2021
Social media is the most commonly accessed internet content by Indonesian netizens; 97.4% Indonesians access social media while using the internet. In 2016, Facebook was the most commonly used social media format for Indonesian internet users. While many are benefited by the features offered by Facebook, many also use Facebook for things that they are not supposed to, such as sharing hoaxes, either intentionally or unintentionally. This clearly puts Facebook users at a disadvantage, considering the increasing trend of hoax-sharing nowadays. Therefore, this research aims to prevent the spread of hoax through Facebook by analyzing the pattern of how people use Facebook. This pattern is obtained from a survey on 200 samples that use Facebook, chosen by purposive sampling. Using the Support Vector Machine method, an application of the collaboration between mathematics and computer science, the acquired data is used to predict whether or not someone has the potential of spreading hoaxes. Simulation results show that the average of the prediction accuracy of this binary classification problem is 86 percent. Hence, it is hoped that Facebook could prevent the sharing of hoaxes by making use of the results from this research.
@inproceedings{Rampisela2023, author = {Rampisela, Theresia Veronika and Andarlia, Hanna Tiara and Rustam, Zuherman}, doi = {10.1088/1742-6596/1725/1/012019}, issn = {17426596}, issue = {1}, booktitle = {Journal of Physics: Conference Series}, keywords = {Classification,Facebook,Hoax,Support vector machines}, title = {Classification of the likelihood of Indonesian Facebook users in spreading hoaxes using Support Vector Machine (SVM)}, volume = {1725}, year = {2021}, month = {}, }
2020
- Academic Expert Finding in Indonesia using Word Embedding and Document Embedding: A Case Study of Fasilkom UITheresia Veronika Rampisela and Evi YuliantiIn 2020 8th International Conference on Information and Communication Technology, ICoICT 2020, 2020
Expertise retrieval covers the problems of expert and expertise finding. In academia, expert finding can be beneficial in finding a research partner or a potential thesis supervisor. This research finds the experts in the Faculty of Computer Science in Universitas Indonesia (Fasilkom UI) using the thesis abstract and metadata of Fasilkom UI students. The methods that are used to represent the query and expertise of the lecturers are the combination of word2vec and doc2vec, which are word embedding and document embedding, respectively. Both embeddings are able to model semantic information, which is necessary for solving the problem of vocabulary mismatch in search problems. Our result shows that representing the expertise query with word2vec leads to better performance than using doc2vec. In addition, we also found that generally, the performance of the embedding models is comparable to the standard retrieval model BM25 in retrieving experts using expertise queries in both Indonesian and English languages.
@inproceedings{Rampisela2020, author = {Rampisela, Theresia Veronika and Yulianti, Evi}, doi = {10.1109/ICoICT49345.2020.9166249}, isbn = {9781728161426}, booktitle = {2020 8th International Conference on Information and Communication Technology, ICoICT 2020}, keywords = {academic expert,document embedding,expert finding,expertise retrieval,word embedding}, title = {Academic Expert Finding in Indonesia using Word Embedding and Document Embedding: A Case Study of Fasilkom UI}, year = {2020}, month = {}, } - Characteristics of Expertise Locator System in Academia: A Systematic Literature ReviewTheresia Veronika Rampisela, Damayanti Elisabeth, and Dana Indra SensuseIn Proceedings - 2020 International Seminar on Intelligent Technology and Its Application: Humanification of Reliable Intelligent Systems, ISITIA 2020, 2020
The ever-increasing need to find experts in academia contributes to the rise of Expertise Locator System (ELS) in academia. ELS can exist in the form of an expert search system or expert recommender system. Albeit the systems’ common purpose of finding an expert, the existing ELS are different from each other in characteristics. There is a consideration, reason, and purpose why an ELS is created with specific characteristics. It is therefore essential to assess and review the characteristics of the system, before creating an ELS. By systematically reviewing recent articles using the Kitchenham method, this research identifies various type of characteristics, so that in the future, ELS in academia can be developed with these characteristics. This study has also identified and proposed seven new characteristics of ELS.
@inproceedings{Rampisela2021, author = {Rampisela, Theresia Veronika and Elisabeth, Damayanti and Sensuse, Dana Indra}, doi = {10.1109/ISITIA49792.2020.9163671}, isbn = {9781728174136}, booktitle = {Proceedings - 2020 International Seminar on Intelligent Technology and Its Application: Humanification of Reliable Intelligent Systems, ISITIA 2020}, keywords = {characteristics,expert finder,expertise locator system,knowledge management,knowledge sharing system}, title = {Characteristics of Expertise Locator System in Academia: A Systematic Literature Review}, year = {2020}, month = {}, } - Expert locator system for finding co-authors using snowball network method: A case study of Fasilkom UI lecturersTheresia Veronika Rampisela, Dana Indra Sensuse, Damayanti Elisabeth, and Nurma Ayu W. SubrotoIn Proceedings of the 4th International Conference on Education and Multimedia Technology, Kyoto, Japan, 2020
One of the essential activities for academics is research. Research is typically done through collaborations. One way to find research collaborators is through existing connections, for example, via the co-author relationships. However, existing systems only provide limited information on collaborators. Furthermore, previous works have not fully explored the details of co-author relationships. Hence, we propose an Expert Locator System (ELS) that can identify collaborators and provide information on the location, publication keywords, as well as their impact and productivity. We can obtain the relationship using the snowball network method, a type of Social Network Analysis (SNA). This ELS can serve to obtain collaborators’ information that can be useful in conducting research.
@inproceedings{Rampisela2022, author = {Rampisela, Theresia Veronika and Sensuse, Dana Indra and Elisabeth, Damayanti and Subroto, Nurma Ayu W.}, doi = {10.1145/3416797.3416831}, isbn = {9781450388375}, keywords = {publication keyword, knowledge sharing system, knowledge management, h-index, expert locator system}, title = {Expert locator system for finding co-authors using snowball network method: A case study of Fasilkom UI lecturers}, year = {2020}, month = {}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3416797.3416831}, booktitle = {Proceedings of the 4th International Conference on Education and Multimedia Technology}, pages = {6–13}, numpages = {8}, location = {Kyoto, Japan}, series = {ICEMT '20}, } - Semantic-Based Query Expansion for Academic Expert FindingTheresia Veronika Rampisela and Evi YuliantiIn 2020 International Conference on Asian Language Processing, IALP 2020, 2020
Expert finding in academic domain is useful for many purposes, such as: to find research collaborators, article reviewers, thesis advisors, thesis examiners, etc. This work examines the use of semantic information, i.e. word embedding and document embedding, for query expansion to enhance the effectiveness of expert finding system. This information is utilized to bridge the lexical gap between the query and the expertise evidence of the experts. This semantic-based query expansion approach is then combined with a BM25 retrieval method to find relevant experts to the given query. The results show that our methods consistently outperform the strong retrieval method BM25, the semantic-based retrieval, and query expansion using pseudo relevance feedback method according to all recall- and precision-based measures used in this work. This indicates the effectiveness of our methods in improving the number and the accuracy of relevant experts retrieved.
@inproceedings{Rampisela2024, author = {Rampisela, Theresia Veronika and Yulianti, Evi}, doi = {10.1109/IALP51396.2020.9310492}, isbn = {9781728176895}, booktitle = {2020 International Conference on Asian Language Processing, IALP 2020}, title = {Semantic-Based Query Expansion for Academic Expert Finding}, year = {2020}, month = {}, pages = {34-39}, }
2018
- Classification of Schizophrenia Data Using Support Vector Machine (SVM)Theresia Veronika Rampisela and Zuherman RustamIn Journal of Physics: Conference Series, 2018
Schizophrenia is a severe and chronic mental disorder. This disorder is marked with disturbances in thoughts, perceptions, and behaviours. Due to these disturbances that can trigger Schizophrenics to commit suicide or attempt to do so, Schizophrenics have a lower life expectancy than the general population. Schizophrenia is also difficult to diagnose as there is no physical test to diagnose it yet and its symptoms are very similar to several other mental disorders. Using Northwestern University Schizophrenia Data, this research aims to distinguish people who are Schizophrenics and people who are not. The data consists of 392 observations and 65 variables that are demographic data and clinician-filled Scale for the Assessment of Positive and Negative Symptoms questionnaires. Classification method used is machine learning with Support Vector Machines (SVM). Simulations are done with different data and percentage of training data. In each simulation, accuracy is measured. Model performance validation and evaluation are done by averaging ten times Hold-Out Validations that were done. In conclusion, SVM successfully classified Schizophrenia data with final accuracy of 90.1%. Furthermore, SVM with linear kernel and Gaussian kernel reached an accuracy of 95.0% in at least one simulation in classifying Schizophrenia data.
@inproceedings{Rampisela2018, author = {Rampisela, Theresia Veronika and Rustam, Zuherman}, doi = {10.1088/1742-6596/1108/1/012044}, issn = {17426596}, issue = {1}, booktitle = {Journal of Physics: Conference Series}, title = {Classification of Schizophrenia Data Using Support Vector Machine (SVM)}, volume = {1108}, year = {2018}, month = {}, }