Evaluation of user model effectiveness by simulation

Accurate initialization of a User Model (UM) is important for every system that provides personalized services. However, there are systems where this initialization is critical yet no user data is available from prior interactions.
For example, systems in which both the repeat usage by a user is rare, and the
total interaction on a single use is relatively limited. Evaluating the effectiveness of a User Model and a particular instance of such a model is never an easy task, particularly through user studies. Moreover, evaluation typically focuses on the usability of an entire system rather than the performance of a specific UM instantiation. In this paper we propose evaluating the quality of UMs via simulation and comparison to a "Gold standard"; this standard is an approximation of a user's ideal model. We will demonstrate this through a case
study of a museum's visitor guide system implemented in the Hecht Museum at
the University of Haifa, Israel