Theresia Veronika Rampisela

Relevance and fairness are two major objectives of recommender systems (RSs). Recent work proposes measures of RS fairness that are either independent from relevance (fairness-only) or conditioned on relevance (joint measures). While fairness-only measures have been studied extensively, we look into whether joint measures can be trusted. We collect all joint evaluation measures of RS relevance and fairness, and ask: How much do they agree with each other? To what extent do they agree with relevance/fairness measures? How sensitive are they to changes in rank position, or to increasingly fair and relevant recommendations? We eempirically study for the first time the behaviour of these measures across 4 real-world datasets and 4 recommenders. We find that most of these measures: i) correlate weakly with one another and even contradict each other at times; ii) are less sensitive to rank position changes than relevance- and fairness-only measures, meaning that they are less granular than traditional RS measures; and iii) tend to compress scores at the low end of their range, meaning that they are not very expressive. We counter the above limitations with a set of guidelines on the appropriate usage of such measures, i.e., they should be used with caution due to their tendency to contradict each other and of having a very small empirical range.

Fairness is an emerging and challenging topic in recommender systems. In recent years, various ways of evaluating and therefore improving fairness have emerged. In this study, we examine existing evaluation measures of fairness in recommender systems. Specifically, we focus solely on exposure-based fairness measures of individual items that aim to quantify the disparity in how individual items are recommended to users, separate from item relevance to users. We gather all such measures and we critically analyse their theoretical properties. We identify a series of limitations in each of them, which collectively may render the affected measures hard or impossible to interpret, to compute, or to use for comparing recommendations. We resolve these limitations by redefining or correcting the affected measures, or we argue why certain limitations cannot be resolved. We further perform a comprehensive empirical analysis of both the original and our corrected versions of these fairness measures, using real-world and synthetic datasets. Our analysis provides novel insights into the relationship between measures based on different fairness concepts, and different levels of measure sensitivity and strictness. We conclude with practical suggestions of which fairness measures should be used and when. Our code is publicly available. To our knowledge, this is the first critical comparison of individual item fairness measures in recommender systems.

Expertise retrieval covers the problems of expert and expertise finding. In academia, expert finding can be beneficial in finding a research partner or a potential thesis supervisor. This research finds the experts in the Faculty of Computer Science in Universitas Indonesia (Fasilkom UI) using the thesis abstract and metadata of Fasilkom UI students. The methods that are used to represent the query and expertise of the lecturers are the combination of word2vec and doc2vec, which are word embedding and document embedding, respectively. Both embeddings are able to model semantic information, which is necessary for solving the problem of vocabulary mismatch in search problems. Our result shows that representing the expertise query with word2vec leads to better performance than using doc2vec. In addition, we also found that generally, the performance of the embedding models is comparable to the standard retrieval model BM25 in retrieving experts using expertise queries in both Indonesian and English languages.

The ever-increasing need to find experts in academia contributes to the rise of Expertise Locator System (ELS) in academia. ELS can exist in the form of an expert search system or expert recommender system. Albeit the systems’ common purpose of finding an expert, the existing ELS are different from each other in characteristics. There is a consideration, reason, and purpose why an ELS is created with specific characteristics. It is therefore essential to assess and review the characteristics of the system, before creating an ELS. By systematically reviewing recent articles using the Kitchenham method, this research identifies various type of characteristics, so that in the future, ELS in academia can be developed with these characteristics. This study has also identified and proposed seven new characteristics of ELS.

One of the essential activities for academics is research. Research is typically done through collaborations. One way to find research collaborators is through existing connections, for example, via the co-author relationships. However, existing systems only provide limited information on collaborators. Furthermore, previous works have not fully explored the details of co-author relationships. Hence, we propose an Expert Locator System (ELS) that can identify collaborators and provide information on the location, publication keywords, as well as their impact and productivity. We can obtain the relationship using the snowball network method, a type of Social Network Analysis (SNA). This ELS can serve to obtain collaborators’ information that can be useful in conducting research.

Expert finding in academic domain is useful for many purposes, such as: to find research collaborators, article reviewers, thesis advisors, thesis examiners, etc. This work examines the use of semantic information, i.e. word embedding and document embedding, for query expansion to enhance the effectiveness of expert finding system. This information is utilized to bridge the lexical gap between the query and the expertise evidence of the experts. This semantic-based query expansion approach is then combined with a BM25 retrieval method to find relevant experts to the given query. The results show that our methods consistently outperform the strong retrieval method BM25, the semantic-based retrieval, and query expansion using pseudo relevance feedback method according to all recall- and precision-based measures used in this work. This indicates the effectiveness of our methods in improving the number and the accuracy of relevant experts retrieved.