Meta-evaluating the effects of social preferences on NPC-evaluators in an energy community game

Energy Communities (EC) are emerging frameworks where citizens collectively share renewable energy. Levering knowledge about this topic is challenging for how varied these types of com munities might be and how many actors are involved in decision taking. We are developing En-join, a game in which the player has to solve open-ended challenges that are mediated and evaluated by conversational agents that represent members of a EC. We im plemented and prompted an LLM (Phi-4) to perform role-playing and evaluation simultaneously. We tested prompt variants indicat ing personality and behavior and meta-evaluated the evaluation performance using six predefined answers across three levels. Our results suggest that indicating social preferences noticeably affects the evaluation behavior. We contribute to the field of games and serious games by showing how LLMs can be used as conversational characters and evaluator agents simultaneously, and suggest that role-playing might be affecting evaluation behavior in any LLM implementations.

Palavras-chave

Large Language Models (LLM) Energy communities Environmental games Serious games . Faculdade de Ciências Exatas e da Engenharia

URI

http://hdl.handle.net/10400.13/7899

Citação

Andrés Isaza-Giraldo, Paulo Bala, Anna Jiskrová, Luiz Sachser, Pedro Campos, and Lucas Pereira. 2025. Meta-evaluating the Effects of Social Pref erences on NPC-evaluators in an Energy Community Game. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25), April 26–May 01, 2025, Yokohama, Japan. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3706599.3720218