What should we evaluate when we use technology in education?

Background: There is a lack of critical or empirical work interrogating the nature and purpose of evaluating technology use in education. Objectives: In this study, we examine the values underpinning the evaluation of technology use in education through field specialist perceptions. The study also poses critical reflections about the rigour of evaluation instruments development in the educational technology field. Method: A total of 48 domain specialists were surveyed to investigate the face and content validity of 39 items under eight constructs, with both qualitative and quantitative data from the survey analysed. Results: There was an alignment between the constructs that the specialists felt were important and the constructs that have typically been the focus of empirical studies, with field specialists indicating high relevance scores for technology (M = 3.24/4), learning outcomes (M = 3.20/4), affective elements (M = 3.19/4), behaviour (M = 3.15/4), presence/community (M = 3.07/4), teaching/pedagogy (M = 3.01/4), design (M = 2.96/4), and institutional environment (M = 2.86/4). Only a minority of other studies were found to perform face and content validity checks and even then only with small samples of respondents (usually n ≤ 5). Implications: Specialists in educational technology research confirm that all eight dimensions are important to consider when evaluating the use of technology in education. Thorough face and content validity processes should be adopted when developing educational technology evaluation instruments. Further work has validated an eight factor 28 item instrument for evaluating the use of technology in education using a large sample of students from a global open learning online course.