Working Paper

Evaluating rater trainings with double-pretest one-posttest designs: An
analysis of testing effects and the moderating role of rater self-efficacy.
Klaus Moser
University of Erlangen-Nürnberg
Verena Kemter
University of Erlangen-Nuremberg
Kerstin Wachsmann
University of Erlangen-Nuremberg
Nora Z. Köver
University of Erlangen-Nuremberg
Roman Soucek
University of Erlangen-Nürnberg
(December 2016)
LASER Discussion Papers - Paper No. 1
(edited by A. Abele-Brehm, R.T. Riphahn, K. Moser and C. Schnabel)
Correspondence to:
Prof. Dr. Klaus Moser, Lange Gasse 20, 90403 Nuremberg, Germany, Email: [email protected].
Abstract
Previous research on performance appraisal in Human Resource Management has found that
frame-of-reference (FOR) training can improve rating accuracy. However, both the time-consuming
development of training materials and the almost exclusive use of experimental evaluation designs
limit the dissemination of FOR training as well as a better understanding of how and when it works.
Therefore, unlike past research, the present studies used more general rater training materials and
examined improvements in rater accuracy by means of a double-pretest one-posttest design. Study 1,
using a student sample (the majority being part-time employees; N = 58), demonstrated the
effectiveness of the rater training over and above a testing effect. Study 2 with participants from the
workforce population (N = 45) replicated these results. In addition, in Study 2, pre-training rater
self-efficacy moderated rater training effectiveness such that the rating accuracy of trainees with low
self-efficacy improved more than did accuracy scores of trainees high in rater self-efficacy. We
conclude that an effective FOR training can be conducted with less organization-specific materials,
though both testing effects must be controlled for and individual differences (i.e. rater self-efficacy)
between trainees have to be taken into account.
Zusammenfassung
Bisherige Forschung hat gezeigt, dass Leistungsbeurteilungen durch Bezugsrahmentrainings deutlich
verbessert werden können. Im Unterschied zu bisherigen Untersuchungen, die vor allem auf
Unterschiede zwischen trainierten und untrainierten Beurteiler konzentrierten (= experimentelles
Evaluationsdesign), analysiert die vorliegende Studie Verbesserungen der Akkuratheit auf der
Grundlage des Vergleichs von Vortest- und Nachtestergebnissen innerhalb eines
Doppel-Pretest-Ein-Posttest-Designs. Es fanden zwei Trainingssitzungen statt. Die Zweite Sitzung
fand eine Woche nach der ertsten Sitzung statt und beinhaltete den zweiten Pretest, das eigentliche
Training, und die Nachtesturteile. Die wahren Werte und die Prinzipien, die die Grundlage für
das Training bildeten, beruhten auf Urteilen von vier Expertenteams.
Zur Evaluation des Trainings wurden diverse Akkuratheitsmaße berechnet (Cronbachs 1955
Akkuratheitskomponenten, D-Index, Distance Accuracy). Erwartungsgemäß zeigten sich bei der
Posttest-Akkuratheit besondere bei den wichtigsten Akkuratheitskomponenten (Cronbachs Differential
Accuracy und Stereotype Accuracy), wobei sich überraschenderweise die Akkuratheit vom ersten
zum zweiten Pretest zunächst verschlechterte.
Copyright statement
Moser, K., Kemter, V., Wachsmann, K., Köver, N. Z., & Soucek, R. (in press). Evaluating rater
trainings with double-pretest one-posttest designs: An analysis of testing effects and the moderating
role of rater self-efficacy. The International Journal of Human Resource Management.
doi:10.1080/09585192.2016.1254102
Author note
Writing of this paper has been partially supported by a grant from the Deutsche
Forschungsgemeinschaft to the second author.