A study of raters’ background knowledge, rater training and other factors affecting their decision making in rating Thai plots’ English speaking proficiency

Abstract

The objectives of this study were to examine two kinds of raters having different background knowledge i.e. linguistic and operational raters with and without rater training experience when they assessed Thai pilots’ English language speaking performances on RELTA, and to explore the other factors affecting their decisionmaking in awarding the scores to the candidates. The participants in the study were 20 raters. They were categorized into two main groups, linguistic and operational, based on their educational and professional background. Then, they were divided further into four sub-groups i.e. linguistic/trained, linguistic/untrained, operational/trained, and operational/untrained. The subjects participating in the main study were purposively selected for data analysis. The source of data was the RMIT English Language Test for Aviation (RELTA). The instruments were questionnaires, rater score sheet and remarks, and semi-structured interviews. The research design was the quasiexperimental research. The 2x2 ANOVA and t-test were used to analyze the quantitative data, and the content analysis was used to analyze the qualitative one, and to confirm the results obtained from the quantitative analysis. It was found that both raters’ background and rater training did not significantly affect the raters’ decision-making in rating Thai pilots’ English speaking proficiency, in both main and interaction effects. However, the factor of training seemed to affect more than the factor of background on the raters’ rating scores. The content analysis analyzed 13 factors that might affect the raters’ decision-making. The results revealed that they could be divided into three groups: the group of the factors which had effects on the raters’ decision-making i.e. rating strategies, candidates/test-takers, rating scale and descriptors, personal relationships between raters and candidates, cut-off score, and scoring: the group of the factors which had no effect on the raters’ decision-making i.e. physical settings, and interviewer/interlocutor: and the group of the factors which were not obvious, hence, unable to make a conclusion i.e. rater educational and rating background, rater mental conditions, rater’s physical conditions, test tasks and speech samples, and raters’ harshness/leniency.