Development of methodologies to evaluate the performance scores of university instructors : a case study in two universities

Abstract

The objectives of the research were to develop a methodology for measuring university instructors' performance scores and to develop formulae for calculating performance change scores over a 3 year period. Following a literature review and interviews with senior academic staff from Thammasat University and Saint John's University, 35 subtasks, grouped under six principal tasks, were proposed for the scoring of university instructors' performance. The subtasks were endorsed as being appropriate by further samples of senior and experienced staff from both universities. Additional samples of senior staff were consulted to determine the weight each task and subtask should contribute to instructor performance scores at each university. Many of the tasks and subtasks had similar weightings at both universities and the differences that did exist, whilst large enough to justify the use of separate sets of weighting factors, tended to follow the pattern that was expected given the two universities' different missions and backgrounds. Data were collected for three academic years (1998, 1999 and 2000) for a total of 290 instructors from both universities. Data for 14 of the subtasks was obtained from existing university documents; a self-report questionnaire was developed for instructors to give data for the remaining 21 subtasks. Cronbach's alpha was around 0.8 each year for both universities, indicating the scores had acceptable internal reliability. When subtasks were ranked by their correlation with total score, there was a high degree of similarity in the rankings each year within each university, implying considerable stability. There was much less similarity in the ranking between the universities, implying that there were differences between the universities in the relationships between the subtasks and the total, even though the reliability of the scores was acceptable at both universities. Samples of Heads of Department from both universities were asked to state whether the scores for each of the instructors within their department were valid. For both universities, 90% of the instructors considered had scores declared valid. The methodology therefore produced performance scores that were reliable and valid within each university. Two formulae were developed for assigning scores to instructors based on the size and pattern of the change in their performance scores over the 3 year period. One formula was developed for linear change, the other for non-linear change. Since almost all the instructors (96%) for whom data were collected exhibited non-linear change, only the non-linear formula could be evaluated. Performance change scores calculated using the researcher's formula were validated by calculating the correlation between them and change scores obtained from the Hierarchical Linear Model (application of quadratic growth model) of Bryk & Raudenbush. The correlation was 0.98 for both universities. Further validation was obtained by asking heads of department and instructors to compare the change scores obtained from the researcher's formula with those obtained from Bryk & Raudenbush's method. 65% of the instructors considered their score from the researcher's formula to be better than that from Bryk & Raudenbush's formula, compared with 10% who thought Bryk and Raudenbush's better. Heads of department considered 62.5% of the researcher's scores to be better, and only 20% of Bryk and Raudenbush's to be better. There was therefore evidence that change scores calculated using the researcher's non-linear formula were valid.