Mettl’s assessments have been the biggest filter in our recruitment process. Their platform has helped us reach out to a higher volume our applicant numbers. Mettl constantly keeps innovating on their products and tries to introduce a new aspect to everything.
Training investments have witnessed a sizable uptick in recent decades. In 2020, the total training expenditure in the USA was approximately USD82.5billion, which is a considerable sum! To put this into perspective, the annual profits of many large corporations are shy of that number. Of course, the numbers may have plummeted temporarily in the wake of the COVID-19 pandemic. Still, training and development initiatives are deemed crucial for organizational growth and workforce development. Organizations cannot benefit from the heady notions of workforce development and continuous learning unless there is compelling evidence of the training programs’ effectiveness.
Training is an essential investment and a critical component of human resource development and, if done well, can significantly boost employee satisfaction, desired outcomes, and organizational effectiveness. However, training efficacy depends on its implementation, and a training program would be baseless if its impact is not taken into account. It is not about the amount of money spent on training; what counts is how relevant training is and how much employees will benefit. That is why evaluating training effectiveness establishes how useful a training program is and what can be done to improve it in the future. One of the methods used to measure training effectiveness is the Kirkpatrick model. This training effectiveness model is best characterized by its easy applicability, the scope of measuring an array of variables, and high usability.
The importance of training and development function in an organization is paramount. A training objective will be fulfilled when employees gain new skills and apply them in their jobs. However, it is not an easy task to evaluate training effectiveness because of the underlying complexity in assessing training effectiveness in terms of ROI (returns on investment). That’s why some proven methodologies are needed to measure and improve the impact of training; such methods are called Training Effectiveness Models.
There are dozens of Training Effectiveness Models currently in use, such as the Kirkpatrick Taxonomy, The Phillips ROI Model, The CIPP Evaluation Model, Brinkerhoff’s Success Case Method, and many more besides. However, in this blog, we will delve more deeply into the Kirkpatrick Evaluation Model, its applications and situations where it is better not to use it.
The Kirkpatrick Evaluation Model is a widely recognized approach to analyze and assess the training program outcomes. It is not only specific to training and educational programs but also includes any program evaluation within its ambit. The model considers both formal and informal training styles to help training practitioners objectively analyze learning outcomes, enabling them to understand the impact of training, how much the trainees have learned, and improve training outcomes. The Kirkpatrick model is Dr. Donald Kirkpatrick’s brainchild, a professor emeritus at the University of Wisconsin.
Although the model underwent several revisions following its first release, the model, as it stands today, is considered the gold standard for assessing business training by learning and development professionals worldwide. The model is suitable to be implemented before, in the course of, and after training to understand the amount of improvement pertaining to training.
The Kirkpatrick Model comprises four distinct levels through which program managers can design training programs and ascertain their effectiveness: reaction, learning, behavior, results.
According to the Kirkpatrick model, a training assessment should always commence at level one, and consequently, time and resource permitting, move subsequently through the remaining levels. The information assimilated from each prior level provides a basis for evaluating the next level. Each successive level specifies a well-defined measure of training effectiveness but simultaneously requires a meticulous and extensive analysis. The complexity of evaluation techniques increases with each level and the data obtained becomes invaluable. It is not surprising that many training experts and departments limit their measurement efforts to levels 1 and 2, owing to the complexity evident in levels 3 and 4 in the Kirkpatrick model. Consequently, the most valuable data remains untapped, which can impact training evaluation efforts.
One must know whether the training delivered to the team has been beneficial. Are employees implementing what they have learned? Does it positively impact business results, such as quality, efficiency, productivity, and customer satisfaction ratings? The Kirkpatrick evaluation model helps provide answers to the questions mentioned above. The model is a sure-fire way to objectively evaluate the training’s effect, determine how much the trainees have learned, and define the scope for improvement.
Now, let us take an in-depth look at each level:
This level aims to identify the participants’ satisfaction levels vis-a-vis the training. Learning and development (L&D) professionals use this level to ascertain how engrossed the trainees were, how they contributed and responded to the training, and how they reacted to the training they received? Was it beneficial? This step can also help in significantly improving future programs by identifying crucial topics that might have been left out.
The participants’ reaction is also measurable through quantitative data collection tools, such as surveys and questionnaires, using rating scales.
An example of key performance indicators:
The second level involves measuring what the participants gained from the training. It gauges the degree to which the trainees assimilated the intended skills, attributes, and knowledge through the training. Instructors and trainers can use this level to ascertain whether they accomplished their goals. Companies can improve upon only by measuring the intensity of learning and determining what participants have and have not learned. Supervisors can test participants in critical areas before and after the training to evaluate the extent of their understanding. This mechanism helps determine what training has notably contributed to enhancing the participants’ knowledge and skills.
Moreover, reviews from trainers and peers can also benefit the assessment at this level.
Quantitative data analysis can be conducted through tests and assessments, after the training, at specific intervals.
An example of key performance indicators:
The next level helps ascertain whether the new learning has catalyzed a behavioral change in the participant. Compared to the previous two levels, this level demands much greater participation and effort from line managers. The participants’ behavioral evaluation is possible by monitoring and interviewing them over time to measure behavior change, how substantial that change is and whether it is sustained. This level involves carefully considering the transfer of learning from the training program to the working environment. It also includes analyzing the work performance of the participants and discerning whether they are making the most of their training on the job.
Measuring the behavior in an effective manner is not a one-off event but a continual process that spans weeks or months following the training session. It includes evaluation through reviews and observations before and after the training.
One can conduct quantitative data analysis through third-party observation, analyzing email conversations of participants and text mining.
In this stage, providing trainees with a conducive work environment can easily ensure that they can apply their new skills. On the other hand, employees would revert to their previous state if the management did not support or appreciate the changed behavior.
An example of key performance indicators:
Level 4 aims to understand how successful the program has been in driving tangible results in the business. It is usually regarded as the crux and the program’s ultimate objective. In this stage, it is common to analyze metrics crucial for establishing training outcomes, such as improved efficiency, better productivity, employee retention, increased sales and higher employee motivation, reduced expenditure, customer satisfaction, etc. It is to be noted that level 4 is a tricky, time-consuming, and costly process because it is not easy to categorize which outcomes, results and benefits can be attributed to training- which is why program managers strive to formulate an effective way to measure training outcomes in the longer run.
Modern trainers now tend to use the Kirkpatrick Evaluation Model backward by focusing on the results they want to achieve and then designing the program to achieve them.
At this level, one can conduct qualitative data analysis through focus groups or interviews. For example, customers may be invited to join a focus group to ascertain whether newly trained employees deliver a superior customer experience, enabling managers to measure the extent to which customer experience has improved over time. Moreover, one can approach the managers to gain insights into whether employees have become more efficient or have improved the quality of their work since the training period.
One can undertake quantitative data analysis through surveys to understand stakeholders’ and customers’ perceptions, compare pre-training data with post-training to assess sales and profits and evaluate the overall employee turnover and retention rate.
An example of key performance indicators:
We have, thus far, deep-dived into each level of the Kirkpatrick Evaluation Model. We can now consider a real-life scenario where the framework would help conduct training evaluation.
The four levels of the training effectiveness model are the pillars that lay the foundation of any successful training program. But the question is: how to implement the model in real life? One of the most recent approaches to using this model is to employ it in reverse, starting with the fourth level and concluding with the first level.
Consider this: A large construction company actively seeks to reduce workplace accidents and create a safe working environment for employees. It wishes to reduce occupational hazards by 25%. A novice training practitioner may commence by developing a training program, but a seasoned training evaluator would take a step back and plan the chronology meticulously.
She or he would first have to define the result, break it down to the desired behavior that will yield results, translate them into learning goals, and then gauge the participants’ reaction to the training.
Here’s how trainers can prioritize the training objectives and increase the training effectiveness using the model backward:
Training providers can plan in reverse and focus first on achieving the desired result, i.e., reducing workplace accidents by 25%.
The only possible means to reducing occupational accidents is ensuring that the floor staff follows all safety protocols each day before beginning work. Trainers have identified the desired behavior that participants would need to exhibit.
What knowledge and training do the floor staff need to learn? By assessing their current awareness levels using pre-tests and surveys, trainers can specify the learning objectives and create a training program accordingly.
Finally, it boils down to how should one develop and deliver this training program to ensure that the trainees (floor staff) enjoy it, correlate it with their jobs and feel fulfilled and empowered once the training is complete? This level aims at determining the participants’ reaction to the training program.
The example mentioned above clearly states that training should always be tied to a business objective and focused on the desired outcome (Level 4). Usually, the value of training is not easily quantifiable, which somewhat complicates outlining its ROI. However, applying the Kirkpatrick model of training effectiveness, particularly in the reverse order, can help managers set their L&D programs for success in the longer run.
Now, let us look at the advantages of the Kirkpatrick Model at different levels in more detail:
The Kirkpatrick Model of Training Evaluation is a widely used tool, but one should use it judiciously. Amid a radically altered world of work, the learning and development ecosystem has undergone dramatic changes. Thus, implementing a six-decade-old model without the proper context can pose significant challenges.
Today, there is no dearth of non-formal workplace training methods that have gained steady ground in the pantheon of the training ecosystem. User-directed and personalized training methods have also come to the fore.
Even though levels 3 and 4 are incredibly crucial metrics for the business, these are equally difficult to implement. So, the Kirkpatrick evaluation framework may not be ideal for organizations lacking a dedicated HR or training department to facilitate the training program.
Most importantly, changing organizational dynamics poses an impact on behaviors, results, and training. For example, significant improvements in productivity and retention could result from a leadership transition, a new technological intervention, rather than simply training.
The Kirkpatrick Model usually measures the effectiveness of a training program through survey-based outcomes. However, organizations are moving toward assessment-based analysis to make the program more robust and bias-free. Assessments provide a means of evaluating training effectiveness using evaluation results and performance data. This evaluation is done via pre-and-post-assessments. Employees are assessed pre-training and post-training, and the evaluation outcomes are compared. The efficacy of the training program is then measured based on the improvement demonstrated by the employee, after the training, in his/her assessment scores. Well-structured assessments enable organizations to assess their workforce’s performance and identify whether the training objectives have been met.
For that reason, Mercer | Mettl’s pre and post-assessment framework is a comprehensive tool that organizations can leverage to measure the performance of online training programs. In addition, this framework enables a seamless and easy measure of the ROI of any training program.
Mercer | Mettl’s pre-training assessments are designed based on a specific competency framework. These assessments, administered before training, measure the current proficiency level of candidates in desired competencies.
Mercer | Mettl’s post-training assessment framework is based on Kirkpatrick’s Level-4 training evaluation model, the global standard for evaluating training effectiveness.
There are countless other models on the market, and it is debatable whether the Kirkpatrick model of training effectiveness is the best means to measure training effectiveness. Every model has its pros and cons. At the end of a training program, what matters is not the model but its execution. The Kirkpatrick Model will mostly work fine. However, it comes down to understanding the outcomes we intend to achieve and then implementing the model correctly.
Well, that is the long and short of it!
We hope this blog has provided you with a comprehensive overview of Kirkpatrick’s four levels of training evaluation and how they can be implemented in designing training interventions.
Training evaluation models are proven frameworks to measure and improve the impact of training.
The Kirkpatrick evaluation model helps impart skill development training by leveraging data from all four levels of measurement. The model is structured to help stakeholders understand how training can be assessed and help focus them on critical performance metrics (such as on-the-job behaviors) alongside other metrics (such as knowledge acquisition and course satisfaction).
For all evaluation purposes, you can use the model in reverse order by beginning with the results stage (level: 4) and working all the way down to the reaction stage (level: 1).
In its pristine form, the Kirkpatrick model follows an order in which evaluation starts from level 1, after which, as time and budget will permit, should proceed through levels two, three, and four. Data acquired from all the levels can be cumulatively analyzed to assess training effectiveness.
Originally published May 17 2021, Updated January 6 2022
Abhilash works with the Content Marketing team of Mercer|Mettl. He has been contributing his bit to the world of online business for some years now. Abhilash is experienced in content marketing, along with SEO. He’s fond of writing useful posts, helping people, traveling, and savoring delicacies.
Training effectiveness is a method to measure the effectiveness of an organization’s training initiatives. Training effectiveness can be determined by qualitative assessments that evaluate the improvement in a trainee’s knowledge, skills and behavior. It can also be quantified in terms of return on investment (ROI).