Theresia Veronika Rampisela

Fairness and relevance are two important aspects of recommender systems (RSs). Typically, they are evaluated either (i) separately by individual measures of fairness and relevance, or (ii) jointly using a single measure that accounts for fairness with respect to relevance. However, approach (i) often does not provide a reliable joint estimate of the goodness of the models, as it has two different best models: one for fairness and another for relevance. Approach (ii) is also problematic because these measures tend to be ad-hoc and do not relate well to traditional relevance measures, like NDCG. Motivated by this, we present a new approach for jointly evaluating fairness and relevance in RSs: Distance to Pareto Frontier (DPFR). Given some user-item interaction data, we compute their Pareto frontier for a pair of existing relevance and fairness measures, and then use the distance from the frontier as a measure of the jointly achievable fairness and relevance. Our approach is modular and intuitive as it can be computed with existing measures. Experiments with 4 RS models, 3 re-ranking strategies, and 6 datasets show that existing metrics have inconsistent associations with our Pareto-optimal solution, making DPFR a more robust and theoretically well-founded joint measure for assessing fairness and relevance. Our code: https://github.com/theresiavr/DPFR-recsys-evaluation

Relevance and fairness are two major objectives of recommender systems (RSs). Recent work proposes measures of RS fairness that are either independent from relevance (fairness-only) or conditioned on relevance (joint measures). While fairness-only measures have been studied extensively, we look into whether joint measures can be trusted. We collect all joint evaluation measures of RS relevance and fairness, and ask: How much do they agree with each other? To what extent do they agree with relevance/fairness measures? How sensitive are they to changes in rank position, or to increasingly fair and relevant recommendations? We eempirically study for the first time the behaviour of these measures across 4 real-world datasets and 4 recommenders. We find that most of these measures: i) correlate weakly with one another and even contradict each other at times; ii) are less sensitive to rank position changes than relevance- and fairness-only measures, meaning that they are less granular than traditional RS measures; and iii) tend to compress scores at the low end of their range, meaning that they are not very expressive. We counter the above limitations with a set of guidelines on the appropriate usage of such measures, i.e., they should be used with caution due to their tendency to contradict each other and of having a very small empirical range.

Fairness is an emerging and challenging topic in recommender systems. In recent years, various ways of evaluating and therefore improving fairness have emerged. In this study, we examine existing evaluation measures of fairness in recommender systems. Specifically, we focus solely on exposure-based fairness measures of individual items that aim at quantifying the disparity in how individual items are recommended to users, separate from item relevance to users. We gather all such measures and we critically analyse their theoretical properties. We identify a series of limitations in each of them, which collectively may render the affected measures hard or impossible to interpret, to compute, or to use for comparing recommendations. We resolve these limitations by redefining or correcting the affected measures, or we argue why certain limitations cannot be resolved. We further perform a comprehensive empirical analysis of both the original and our corrected versions of these fairness measures, using real-world and synthetic datasets. Our analysis provides novel insights into the relationship between measures based on different fairness concepts, and different levels of measure sensitivity and strictness. We conclude with practical suggestions of which fairness measures should be used and when. Our code is publicly available. To our knowledge, this is the first critical comparison of individual item fairness measures in recommender systems.

Expertise retrieval covers the problems of expert and expertise finding. In academia, expert finding can be beneficial in finding a research partner or a potential thesis supervisor. This research finds the experts in the Faculty of Computer Science in Universitas Indonesia (Fasilkom UI) using the thesis abstract and metadata of Fasilkom UI students. The methods that are used to represent the query and expertise of the lecturers are the combination of word2vec and doc2vec, which are word embedding and document embedding, respectively. Both embeddings are able to model semantic information, which is necessary for solving the problem of vocabulary mismatch in search problems. Our result shows that representing the expertise query with word2vec leads to better performance than using doc2vec. In addition, we also found that generally, the performance of the embedding models is comparable to the standard retrieval model BM25 in retrieving experts using expertise queries in both Indonesian and English languages.

Expert finding in academic domain is useful for many purposes, such as: to find research collaborators, article reviewers, thesis advisors, thesis examiners, etc. This work examines the use of semantic information, i.e. word embedding and document embedding, for query expansion to enhance the effectiveness of expert finding system. This information is utilized to bridge the lexical gap between the query and the expertise evidence of the experts. This semantic-based query expansion approach is then combined with a BM25 retrieval method to find relevant experts to the given query. The results show that our methods consistently outperform the strong retrieval method BM25, the semantic-based retrieval, and query expansion using pseudo relevance feedback method according to all recall- and precision-based measures used in this work. This indicates the effectiveness of our methods in improving the number and the accuracy of relevant experts retrieved.