• reviews the thinking on evaluation of training
• emphasises the need to decide why training and learning are to be evaluated, before moving into how
• provides some straightforward techniques for valuing learning using the latest CIPD model.

The development of thinking on evaluating training

In 1959, Kirkpatrick¹ first outlined four levels for training evaluation:

• reactions – ‘liking or feelings for a programme’
• learning – ‘principles, facts etc absorbed’
• behaviour – ‘using learning on the job’
• results – ‘increased production, reduced costs, etc’.

For the next 45 years the evaluation of training moved on very patchily in terms of research and new ideas, and poorly in terms of practical application. In 2007, however, the CIPD produced a new ‘partnership of learning model’ which emphasises the need for all those involved in learning interventions actively to play their part.

Why is it so important to decide why training and learning are to be evaluated?

The simple answer is that until the purpose of any evaluation is established, the questions to ask, measures to seek, and stakeholder groups to approach cannot be determined. The new CIPD model is rightly about employers, line managers, individual learners and trainers/facilitators making sure that:

• intended learning and organisational strategies are aligned from before the start of any learning programme
• appropriate measures are defined
• skills and organisational performance are raised.

In this context, evaluation is in itself about ensuring relevant, integrated and continuous learning. In other words evaluation is an intervention to check alignment and to reinforce learning – as set out in the Levi Strauss case study³ where the evaluation questions are based on the deployment of relevant skills learned and, powerfully, posed by senior stakeholders in personal calls and interviews or on site visits.

What happened between Kirkpatrick in 1959 and the CIPD in 2007?

In 1989, Alliger & Janak⁴ examined the assumptions which had grown up around what Kirkpatrick had probably intended to be a simple four-stage guide. They questioned whether each level was positively linked to the previous one. Only a small sample of evaluations carried out in the 30 years since 1959 were found to be good enough for analysis, and the results for causality and correlation were not great – not much better than random chance! So, someone could almost equally love or hate a course and learn or not learn from it irrespectively, or could learn a great deal from it but not translate that into improved job or organisational performance.

In contrast to this training-centred approach, in 1989, Guzzo et al⁵ conducted a useful analysis of ‘psychologically-based interventions to raise worker productivity’. 207 studies had to be whittled down to the 98 having enough data and robustness, and findings showed that training and goal setting were best, with ‘management by objectives’ and supervisory methods the worst. Unfortunately, it seems that this sort of study (that is, comparing what actually helps to raise performance, rather than assuming that training alone will achieve it) has not been repeated seriously.

The emergence of four purposes for evaluation

1. Proving the value of training

Stemming from the call for evidence that the US Second World War ‘Training Within Industry Initiative’ was worthwhile, the push to use evaluation for proving purposes has continued to the present day. In the 1970s, the emphasis was mainly on scientifically-based research, a classic case being Latham and Saari’s study of training for factory supervisors⁶. Using sound experimental-and-comparable-control-group processes, they were able to confirm reasonably well that learning had been translated into better capabilities to perform the job, and had brought about a sustained improvement.

In the 1980s, concerns about proving a return on investment (ROI) emerged. Mathieu and Leonard⁷ showed that training for bank staff could be expected, over a 20-year period, to give substantial returns (ranging from US$63,000 for head tellers to US$255,000 for branch managers). Later, Morrow et al⁸ arrived at ROI figures for training in a pharmaceutical company. They found that the ROI on technical and sales training was over nine times greater than on management training (but the study took four years and cost US$ 0.5m!).

2. Improving the quality of the training offered

Back in 1963, Cronbach⁹ had introduced the concept of evaluation for the purpose of improving training.

Scriven¹⁰ soon highlighted the formative, action-oriented nature of this sort of evaluation, as compared to proving that training had worked. Nowadays, the accepted wisdom is proving the quality of the training offered should come before assessing the value of any particular course or programme.

3. Evaluation as a contribution to the learning process

Next came the very important strand of evaluation as a contribution to the learning process itself. In 1973, Burgoyne¹¹ applied action learning principles to a business school management programme and saw an increase in participant learning, as perceived by the participants themselves. Ruddock¹² built on the idea of learning from evaluation by introducing the concept of illuminative analysis to increase communal awareness of a programme. In 1967 Scriven¹⁰ had proposed the radical view that an evaluator should take no notice of the (politically-decided) goals of a programme, and should, instead, focus on how the programme impacted on other (receiving) stakeholders.

Guba and Lincoln¹³ took these ideas further, and proposed that, following the first three generations of evaluation (that is, measurement, description, and judgement), a fourth generation was essential. They called this ‘responsive constructivist’, and defined it as a form of negotiation in which the claims, concerns and issues of all stakeholders should serve as the focus for the meaning and implications of any programme and its evaluation.

More recently, Silvester et al¹⁴ partially used this sort of thinking in an evaluation of a culture change programme in a large engineering company. They found that three stakeholder groups – managers, trainers and trainees – all considered that the programme would produce positive outcomes, but also that:

• the trainees were the most optimistic about this – perhaps because of receiving attention and the chance to learn new skills (though not necessarily to be involved in the apparent push to change the organisational culture)
• the trainers were the most cynical – harbouring doubts about their abilities to bring about lasting change
• the managers the most pessimistic – (believing that the training was about quality not culture, and that it would thrive only locally).

4. Evaluation as control over training

Finally, in terms of the purposes for evaluation, back in 1974, Hamblin¹⁵ had put forward the need to evaluate training to control its costs and access to it. Bramley¹⁶ also showed that, particularly in the public sector, control might be necessary for consistency and compliance.

Pulling all this together involves Patton’s plea for attention to the utilisation of evaluation¹⁷, and Easterby-Smith’s neat encapsulation of the four strands or purposes of evaluation¹⁸:

• proving
• improving
• learning (reinforcing)
• controlling.

Choosing the purpose of evaluation

Consider for a moment that you are an evaluator with limited time and resources. What would your preferences be in terms of allocating your time and effort to one of more of the four purposes (and how would that affect the what, how, when and of whom you ask questions)? Alternatively, if you already have a strong stance about the purpose of training evaluation, how would you deal with a client who came from quite an opposite view?

Where do things stand today?

Exhortations to evaluate training continue to abound, systems are constantly recommended, and still little is done. Concepts like ROI have been strongly recommended, some individual events are tested for continuing validity and effect, and some attention has been paid to distinguishing different stakeholders’ views. But people still rarely pay attention to the purpose of evaluation (as the examination of the general hotch-potch of questions in a typical end-of-course feedback sheet will reveal).

Many ‘thinking practitioner’ trainers/facilitators would strongly advocate dropping what are seen as ‘happy sheets’, in favour of spending more time and effort on follow-up activities to ensure that action plans are implemented, learning is applied and reinforced, and that the training is not seen as an isolated event in space and time. Above all, there seems to be an ‘evaluation lament’ – lack of time, low line manager interest and absence of easy systems are cited all too often as reasons why evaluation is still patchy and poor.

How can the new CIPD model move evaluation forward?

The CIPD model is all about shifting perceptions from learning in isolation to relevant, integrated and continuous learning. The first task is therefore to raise the implications of this shift with key stakeholders in the organisation, and to begin to enhance senior management trust in the learning contribution.

As set out in our Change agenda², it is important to:

• consider the view of managers at all levels about suitable metrics for assessing and reporting on the value of learning
• ensure that all trainers/facilitators understand the business and can identify how to relate any given organisational metrics to learning initiatives.

Both of these tasks may be easier in organisations where ‘hard’ skills training can show immediate results, for example, proving value – as in a sales environment. The tasks may be less easy where ‘soft’ skills are concerned, for example management training, but it is still possible to uncover value through follow-up and discussion, such as in coaching environments.

Recent examples where general management training has been found to have value, through questions designed to reinforce learning, have come from a financial firm (where respondents explained how they had found that the application of new ways of cooperation was leading them to meet cost-reduction metrics), a newspaper group (where respondents were using new ways of thinking to find more revenue) and a government department (where respondents said they were now looking at higher quality problem solving). Interestingly, the Levi Strauss case study³ indicates that learning which affects top-line results will not only be measured by the interested stakeholders but will also mean less need to prove the effectiveness of the training team.

Some practical tips

Once a possible shift from ‘happy sheets’ to follow-ups is accepted, the what and how of information gathering about the value of learning (and improved event design) become much easier.

The ‘what’ must include management information about issues important to the organisation. And the initial question to put to hard-pressed senior managers is: What measurable results (both short-term and long-term) from the learning function would you like to see for your area of responsibility? Some possible measures, related to each of the four main purposes for evaluation, are:

The ‘how’ includes gathering feedback from two major stakeholder groups: learners and their managers. The main ways are:

• questionnaires from the learning function
• interviews and focus groups
• feedback from performance reviews
• one-to-one discussions between managers and their staff
• self-reporting by learners.

Questionnaires generally have a notoriously low return rate, and need to be very carefully thought through and piloted (to check for, amongst other things, question purpose and clarity, and the ease of and time taken for completion). They are, however, useful in large organisations, where interviews and focus groups might be too time consuming.

Interviews and focus groups allow for discussion and real exploration about learning, but they take time and cannot cover many people quickly.

Feedback from performance reviews seems to be a much under-used source of evidence of the value of learning. A few organisations do include a box to cover the use of training and other learning activities, but hardly any take the chance to link the results of such activities to organisational strategies, plans and metrics.

A few, go-ahead organisations have taken up the opportunity to ask managers for feedback about the impact of training and development for their staff along ‘Kirkpatrick’ lines, for example, different behaviour/attitude demonstrated on return from a learning event, evidence of impact on the job, and on the bottom line of the organisation’s business.

Self-reporting by learners has some significant advantages: it encourages people to think about what and how they have learned and are continuing to develop, to retain ownership of and responsibility for their development and growth, and experience shows that many gain pride from reporting successful learning. Problems to overcome, however, include generating the sort of culture that encourages such self-reporting, and ways to collect it consistently (performance review provides one way, others include following up on action plans from learning events and interviews).

Evaluation questions

Finally, some good evaluation questions for reinforcing learning include (with the underlying rational for each question following in brackets):

• What was the most important or valuable learning for you? Why? How did you learn this? (reflecting on the immediate past experience and looking for insightful, long-lasting ‘ah-ha’ moments that help people to see that they can learn, and how they learn)
• How can you apply what you learned to the needs of your job, your team/department, and the priorities of the business? (thinking about the present and looking for wider learning application and its relationship to the business and its chosen metrics)
• How could you integrate what you learned into further learning opportunities available to you, and the developing requirements of your job, your team/department, the organisation as a whole? (pointing to the future and looking for the multiplicative effects of the variety of ways people can learn).

References

1.     KIRKPATRICK, D.L. (1959). Techniques for evaluating training programs. Journal of the American Society of Training and Development. Vol 33, No 11. pp3-9.

2.     ANDERSON, V. (2007) The value of learning: a new model of value and evaluation. Change agenda. London: Chartered Institue of Personnel and Development. Available at: http://www.cipd.co.uk/changeagendas

3.     CHARTERED INSTITUTE OF PERSONNEL AND DEVELOPMENT. (2005) The way to sell: training’s contribution to a business problem at Levi Strauss & Co [online]. London: CIPD. Available at: http://www.cipd.co.uk
/helpingpeoplelearn/_casestudies/_vlvistrss.htm

4.     ALLIGER, G. and JANAK, E. (1989) Kirkpatrick’s levels of training criteria: thirty years later. Personnel Psychology. Vol 42, No 2. pp331-342.

5.     GUZZO, R. and GANNET, B. (1989) The nature of facilitators and inhibitors of effective task performance. In: SCHOORMAN, F. and SCHNEIDER, B. Facilitating work effectiveness. Lexington: Lexington Books.

6.     LATHAM, G. and SAARI, L. (1979) The application of social learning theory to training supervisors through behavioural modelling. Journal of Applied Psychology. Vol 64. pp239-246.

7.     MATHIEU, J. and LEONARD, R. (1987) Applying utility concepts to a training program in supervisory skills: a time-based approach. Journal of Academic Management. Vol 30, No 2. pp316-335.

8.     MORROW, C., JARRET, Q. and RUPINSKI, M. (1997). An investigation of the effect and economic utility of corporate-wide training. Personnel Psychology, Vol 50, No 1. pp91-117.

9.     CRONBACH, L. (1963). Evaluation for course improvement. In: HEATH, R. (ed). New curricula. New York: Harper & Row.

10.  SCRIVEN, M.(1967). The methodology of evaluation. In: AMERICAN EDUCATIONAL RESEARCH ASSOCIATION. Perspectives of curriculum education. Monologue No 1. Chicago: Rand McNally.

11.  BURGOYNE, J. (1973). An action research experiment in the evaluation of a management course. Journal of Management Studies., Vol 10, No 1. pp8-14.

12.  RUDDOCK, R. (1981). Evaluation: a consideration of principles and methods. Dorset: Manchester Monographs.

13.  GUBA, E. and LINCOLN, Y. (1989) Fourth generation evaluation. London: Sage.

14.  SILVESTER, J., ANDERSON, N. and PATTERSON, F. (1999). Organisational culture change: an inter-group attributional analysis. Journal of Occupational and Organisational Psychology, Vol 72, No 1. pp1-23.

15.  HAMBLIN, A. (1974). Evaluation and control of training. Maidenhead: McGraw-Hill.

16.  BRAMLEY, P. (1996). Evaluating training. London: Chartered Institue of Personnel and Development.

17.  PATON, M. (1986). Utilisation-focused evaluation. Beverly Hills: Sage.

18.  EASTERBY-SMITH, M. (1994) Evaluating management development, training and education. Aldershot: Gower.

This factsheet was written by Iain Thomson is a Fellow of the CIPD.